The whole product is a router, a lease table, and a command line tool. This page covers what each part does and where the limits are.
A worker asks for work and receives a lease with an expiry. While the lease is held, no other worker can claim that job. If the worker finishes, it acknowledges and the job is marked complete. If the worker disappears, whether from an out-of-memory kill, a deploy, or a network partition, the lease expires and the job becomes claimable again.
The practical difference from row locking is what happens during an incident. A lock held by a dead connection needs either a timeout you configured months ago or a human with database access. A lease that expires needs neither.
Lease duration is per route. Short leases recover faster from worker loss but require more frequent renewal for long-running jobs. Renewal is a single call, and the client library handles it on a timer.
$ recology lease inspect reports/4c81a
job reports/4c81a
state leased
worker worker-03.internal
acquired 12s ago
expires in 18s
attempts 1 of 5
trace 4c81a9e07b2d
A retry budget is a rate limit on failure, expressed as attempts per unit of time for a whole route rather than per job. When a route exhausts its budget, further retries wait instead of firing. This is the mechanism that keeps a broken downstream dependency from turning into a self-inflicted denial of service.
throttled state and the reason appears in the audit stream.Jobs that exhaust their attempts move to a dead-letter queue on the route rather than being deleted. The original payload, headers, attempt history, and trace identifier are all retained.
Replay reuses the stored payload and trace identifier, so the retried job appears in your tracing backend as another attempt on the same operation. You can replay a single job, a filtered set, or an entire route's dead-letter queue.
Dead-letter retention defaults to 14 days and is configurable per route. Expired entries are pruned on a schedule, and the prune job reports how many entries it removed.
$ recology dlq list webhooks --limit 3
id attempts last error
webhooks/9f21c 5 502 from dispatch.internal
webhooks/9f22a 5 502 from dispatch.internal
webhooks/9f31d 5 context deadline exceeded
$ recology dlq replay webhooks --all --rate 10/s
queued 311 jobs for replay at 10/s
Two surfaces, both read-only. The command line tool is for answering a question right now. The metrics endpoint is for the dashboards and alerts you already maintain.
| Metric | Type | What it tells you |
|---|---|---|
recology_route_pending | gauge | Jobs waiting for a lease, labelled by route. |
recology_route_leased | gauge | Jobs currently held by a worker. |
recology_route_lag_seconds | gauge | Age of the oldest pending job. The number to alert on. |
recology_attempts_total | counter | Attempts, labelled by route and outcome. |
recology_budget_throttled_total | counter | Attempts deferred because a retry budget was spent. |
recology_dlq_depth | gauge | Dead-letter entries per route. |
There is no DAG support. If your work has fan-out, fan-in, and conditional branches, you want a workflow engine, and running one alongside Recology is a reasonable design.
There is no hosted option. The store is your Postgres instance and the workers are your processes. That rules out some convenience, and it also means job payloads never leave your network.
There is no built-in scheduler beyond simple intervals. Cron-style scheduling with timezone handling is a genuinely hard problem and we would rather point you at a tool that has solved it than ship a partial version.
Read the documentation for the parts that do exist, or the changelog for what has moved recently.