Invoices were not generated for a week. The nightly export has been missing since a deployment eleven days ago. Nothing alerted, because monitoring watches for errors and there were none - the job simply never ran. Absence of work generates absence of signal, which is why this class of failure is routinely discovered by a customer rather than by a dashboard.
01Monitor for completion, not for failure
Error-based alerting detects jobs that run and break. It cannot detect jobs that never start, and that is the more common failure: a worker not restarted after a deploy, a schedule removed during a configuration change, or a queue consumer that died and was never replaced.
Dead man's switch monitoring inverts the logic. The job reports success on each run, and the monitor alerts when that report fails to arrive within an expected window. Silence becomes the alert rather than the normal state.
| Failure | Error alerting | Completion monitoring |
|---|---|---|
| Job throws an exception | Detected | Detected |
| Job never scheduled | Missed | Detected |
| Worker process died | Missed | Detected |
| Job hangs indefinitely | Missed | Detected |
| Queue consumer disconnected | Missed | Detected |
| Job completes doing nothing | Missed | Detected if it reports items processed |
02Report what was done, not just that it ran
A job that completes successfully having processed zero records is often broken. A query filter that no longer matches, a date boundary that excludes everything, or an upstream table that stopped being populated all produce a clean exit with no work performed.
Emit the count of items processed and alert on implausible values. A nightly job that normally handles thousands of records and suddenly handles none is a strong signal, and it is only available if the job reports volume rather than status alone.
03Watch the queue, not only the workers
Queue depth is the clearest indicator of whether consumption is keeping pace with production. Steadily growing depth means consumers have stopped or cannot keep up, and it is visible long before the effects reach users.
Oldest message age is the more useful metric of the two. A queue holding a constant thousand messages that are all seconds old is healthy; one holding fifty messages that are six hours old is not, and depth alone cannot distinguish them.
| Signal | Indicates |
|---|---|
| No completion within expected window | Job stopped running |
| Queue depth growing steadily | Consumers stopped or too slow |
| Oldest message age rising | Processing has stalled |
| Dead letter queue non-empty | Messages failing repeatedly |
| Items processed far below normal | Silently doing no work |
| Duration far above normal | Approaching a timeout or overlap |
04Dead letter queues need an owner
A dead letter queue is where messages go after repeated processing failures. It exists so that one poisonous message cannot block a queue indefinitely, which is valuable. What frequently happens is that it accumulates for months and nobody looks, so the failures it captured are never addressed.
Alert on any message arriving in a dead letter queue and treat it as work rather than as an archive. Each entry is a real operation that did not happen - an unsent email, an unprocessed payment, an unsynchronised record.
05Make jobs safe to re-run
Recovery after a gap means re-running for a period, which is only safe if jobs are idempotent. A job that sends emails must not send duplicates when re-run; one that aggregates must not double-count. Designing for that upfront makes recovery routine instead of a manual reconciliation exercise.
Guard against overlap too. When a job runs longer than its interval, a second instance can start alongside the first and both process the same records. A lock, with a timeout so a crashed run does not block all future ones, prevents that.
Topics
Priya Iyer
Staff Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer