Engineers rarely leave because of one terrible night. They leave after months of being woken by alerts that turned out to be nothing, having no documentation for the ones that were something, and watching the same failure recur because fixing it properly was never prioritised. Each of those is addressable, and none of them requires a large team.
01Every page must be actionable
An alert should mean a human needs to do something now. If the correct response is to acknowledge and go back to sleep, it should not have paged - it should be a ticket, or nothing. Alerts that fire regularly without requiring action train people to ignore the channel, and that habit persists into the incident that matters.
Audit alerts quarterly against a simple test: over the last three months, what proportion of firings led to an action? Anything below about eighty percent should be retuned, downgraded or deleted. Deleting alerts feels risky and is usually the single highest-value change available.
| Condition | Route | Rationale |
|---|---|---|
| Users cannot use the product | Page immediately | Genuine emergency |
| Error rate breaching the objective | Page immediately | Users affected now |
| Disk at 80% | Ticket, business hours | Hours of runway remain |
| Certificate expiring in 14 days | Ticket | Nothing to do at 3am |
| A single job failed and retried | Nothing | The system handled it |
02Runbooks are the difference between competence and panic
Every alert needs a linked runbook describing what the alert means, how to confirm the problem is real, the first three things to check, how to mitigate, and who to escalate to. Without one, the person on call is rediscovering the system at three in the morning under time pressure.
Write them as part of creating the alert, not afterwards. An alert without a runbook is an incomplete piece of work, and treating it that way is what stops the backlog of undocumented alerts from accumulating.
03Rotations for teams that are too small
Below about six engineers, a traditional rotation is punishing - one week in three or four is relentless. Options that help: a primary and secondary pair so nobody is alone, business-hours-only coverage for anything that genuinely can wait, and honest compensation whether in money or time off.
The most important structural fix is reducing what pages at all. A team of four with three real pages a month is sustainable. The same team with thirty is not, and no rotation design fixes that - only alert hygiene does.
| Signal | Indicates |
|---|---|
| More than 2-3 pages per week | Alert tuning or real reliability problem |
| Same alert recurring monthly | Nobody is fixing the cause |
| Pages with no runbook | Documentation debt |
| One person handles the hard ones | Knowledge concentration |
| Engineers declining the rotation | The clearest signal available |
04Follow-up is what stops repetition
The work that prevents the next incident happens after this one, and it is the work most reliably deprioritised. Every incident should produce follow-up items with owners and dates, and those items need to compete with feature work rather than sitting in a separate list nobody reads.
Track the proportion completed within a month. If it is low, incidents will keep recurring and on-call will keep degrading, regardless of how good the postmortem documents are.
05Blameless, and specific about it
Postmortems that identify a person as the cause produce engineers who hide mistakes, which makes the next incident longer and harder to diagnose. The useful question is not who ran the command but why the system allowed a single command to cause that outcome.
Blameless does not mean vague. 'The deploy pipeline permits schema changes without review' is a specific, actionable finding. 'Mistakes were made' is neither. The goal is to describe the system honestly enough that the fix is obvious.
Topics
Marcus Hale
Lead Architect · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer