Background jobs that do not lose work: queues, retries and idempotency

01Jobs must be idempotent

Every realistic queue delivers at least once. A worker that crashes after completing the work but before acknowledging will see the message again. Network partitions redeliver. Retries redeliver. If running a job twice produces two charges, two emails or two records, the system is broken and it is only a question of when.

Design for redelivery from the first job you write. Record a stable job identifier and the fact of completion in the same transaction as the effect, then skip work already recorded. Retrofitting this is considerably harder than building it in, because by then the effects are spread across systems you no longer control atomically.

• Give every job a stable, deterministic identifier.
• Record completion in the same transaction as the effect.
• Use upserts rather than inserts where the operation permits.
• For external calls, pass an idempotency key the provider honours.

02Keep payloads small and by-reference

Serialising a large object into the queue message creates two problems. The payload can become stale between enqueue and execution, so the job operates on data that has since changed. And the message may exceed size limits or bloat the queue.

Enqueue identifiers instead. The job fetches current state when it runs, which is almost always what you want - and where you specifically need the state as of enqueue time, capture that explicitly as a version or snapshot reference rather than relying on the payload as an accidental snapshot.

03Retry with the same discipline as network calls

Job retries need exponential backoff and jitter for the same reason API retries do: a downstream dependency that is struggling gets worse if every failed job retries immediately and simultaneously. Failed jobs cluster, so their retries cluster too unless you deliberately spread them.

Distinguish retryable from permanent failures. A timeout calling an external service is worth retrying. A validation error on a malformed record will fail identically forever, and retrying it twenty times just delays discovery and consumes worker capacity. Fail permanent errors immediately to a dead letter queue where they are visible.

04The dead letter queue needs an owner

Every system accumulates jobs that exhausted their retries. Without a dead letter queue they vanish silently. With one that nobody monitors, they accumulate silently - which is only marginally better, because the information exists and nobody reads it.

Alert on the dead letter rate, not just its depth, so a sudden increase is visible immediately. Retain enough context to diagnose and replay: the original payload, the error, the attempt count and timestamps. And build the replay mechanism before you need it, because the moment you need it is during an incident where a dependency was down for an hour.

05Monitor queue depth and age together

Queue depth alone is ambiguous - a thousand messages might be a healthy burst being processed quickly or a stalled worker. The more informative metric is the age of the oldest unprocessed message, which directly answers how far behind you are.

Alert on age crossing a threshold that reflects the business meaning of the work. A password reset email delayed five minutes is a serious problem; an analytics rollup delayed five minutes is fine. Separate queues by urgency so a flood of low-priority work cannot delay time-sensitive jobs behind it, which is one of the most common causes of user-visible async failures.

• Alert on oldest-message age, not only queue depth.
• Separate queues by urgency so slow work cannot block fast work.
• Cap job duration and make long jobs resumable in chunks.
• Track throughput so capacity problems are visible before they bite.