Recology

Scheduling that stays legible at scale

A single binary that routes background work across your own workers, with lease semantics you can reason about and metrics you already know how to scrape.

$ recology route add reports --workers 4 --lease 30s
created route "reports" (lease 30s, retry budget 5/min)

$ recology status
route        pending  leased  failed  lag
reports           12       4       0   1.4s
thumbnails         0       2       0   0.2s
webhooks         311      16       3   8.9s

$ recology tail webhooks --since 5m
# 3 jobs exceeded their retry budget and moved to the
# dead-letter queue. Replay preserves the trace id.
warn  webhooks/9f21c dead-lettered after 5 attempts

Built for services that outgrew cron

Most teams reach for a background job system twice. The first time, a cron entry and a database table are enough. The second time, usually after a retry loop has taken down a downstream service, they need leases, visibility, and a way to answer why a particular job ran three times.

Recology sits in that second slot without asking you to adopt a workflow engine. It is one static binary that runs on your own hosts, keeps its state in Postgres, and exposes a configuration file you can commit alongside your application code. There is no hosted control plane and no agent to install on every machine.

Work is grouped into routes. A route has a concurrency limit, a lease duration, and a retry budget. Workers claim leases rather than locking rows, so a worker that dies mid-job releases its work when the lease expires instead of holding it until someone notices.

Everything a route does is visible from the command line and from a Prometheus-compatible metrics endpoint. If queue depth climbs, you can see which route is responsible and whether the cause is throughput or a downstream failure, without attaching a debugger to a worker.

Configuration is a file, not a dashboard

Route definitions live in a single YAML file. Changes go through the same review process as the rest of your code, which means the answer to "who changed the retry budget" is in your version history rather than in an audit log you have to go and find.

The configuration validates on load. A malformed duration or an unknown target fails at startup with the offending line number, not on the first job that happens to touch it.

# recology.yaml
store: postgres://recology@db.internal:5432/recology

defaults:
  lease: 30s
  backoff: exponential
  jitter: true

routes:
  - name: reports
    workers: 4
    retry_budget: 5/min
    target: http://reports.internal:8080/run

  - name: webhooks
    workers: 16
    retry_budget: 30/min
    dead_letter: true
    target: http://dispatch.internal:8080/send

What you get out of the box

Lease semantics

Jobs are leased, not locked. A crashed worker returns its work automatically when the lease runs out.

Retry budgets

Budgets are per route, so one misbehaving integration cannot spend the whole system's retry capacity.

Readable metrics

Queue depth, lag, and failure counts on a standard metrics endpoint your existing scrape config can pick up.

Dead-letter replay

Failed jobs keep their payload and trace identifier so a replay is a real retry, not a reconstruction.

Single binary

No runtime to install and no sidecar. Copy the binary, point it at Postgres, and run it under systemd.

Structured audit output

Every state transition is emitted as JSON Lines for whatever you already use to store logs.

Recent releases

Recology ships on a roughly monthly cadence. The full history, including breaking changes and migration notes, is on the changelog.

1.10.0

  • Retry budgets are now configurable per route rather than per worker.
  • Backoff is jittered by default to avoid synchronized retry storms.
  • Dead-letter replay now preserves the original trace identifier.

1.9.0

  • Improved error messages when a route target fails DNS resolution.
  • Added a dry-run mode that prints the resolved plan and exits.
  • Added structured audit output in JSON Lines for downstream ingest.