Incident response material is mostly written by and for large organisations with dedicated reliability teams, severity matrices and formal command structures. A team of twelve cannot adopt that and should not try. But a handful of practices from it deliver most of the benefit at almost no overhead, and they are worth having in place before the first serious outage rather than after.
01Someone is in charge
The most valuable practice from formal frameworks is designating one person as incident commander. Not the most senior person, not necessarily the best debugger - the person coordinating. They decide what is being tried, who is trying it, and what gets communicated.
Without this role, small teams reliably fail the same way: five engineers independently investigating, three changing production simultaneously, nobody updating stakeholders, and no shared picture of what has already been ruled out. The commander is often more valuable when they are not the person debugging, because holding the overall picture is a full-time job during an incident.
- Name a commander in the first minutes, explicitly and out loud.
- The commander coordinates rather than debugs.
- One person makes production changes; others propose.
- Keep a running timeline in the incident channel as you go.
02Mitigate before diagnosing
The instinct of good engineers is to understand the problem before acting. During an incident that instinct is expensive. If a deploy thirty minutes ago correlates with the errors, roll it back and investigate afterwards. Understanding why can happen while the system is healthy.
Build the mitigations that make this possible in advance: fast rollback, feature flags that disable a subsystem, the ability to shed load or fail over. A team that can roll back in two minutes has a fundamentally different incident profile from one where rollback means a twenty-minute pipeline, regardless of debugging skill.
03Alert on symptoms, not causes
Alerting on every resource metric produces noise that trains people to ignore alerts. High CPU is not inherently a problem, and an alert that fires regularly without action being needed is worse than no alert, because it erodes trust in all of them.
Alert on things users experience: elevated error rates, latency beyond an acceptable threshold, a critical flow failing, a queue growing without bound. These are few, they are meaningful, and every one deserves a human response. Everything else belongs on a dashboard consulted during investigation rather than in a paging channel.
| Signal | Page? | Rationale |
|---|---|---|
| Error rate above baseline | Yes | Users are affected now |
| Latency p99 past threshold | Yes | Degraded experience, may cascade |
| Checkout or auth flow failing | Yes | Direct revenue or access impact |
| Queue depth growing unbounded | Yes | Predicts imminent failure |
| High CPU | No | Normal under load; dashboard signal |
| Single instance unhealthy | No | Orchestrator should handle it |
| Disk above 70% | No | Ticket; only page near exhaustion |
04Runbooks over documentation
Comprehensive documentation is written once, read never, and becomes wrong within months. A runbook is different: a short, specific procedure for one recurring situation, written immediately after doing it, and it earns its maintenance because it gets used.
Write one after every incident where you had to work something out - how to fail over the database, how to clear the stuck queue, how to rotate the credential that expired. Keep them short and current. A runbook that has drifted from reality is more dangerous than none, so validate them when the underlying system changes.
05Postmortems without blame or bureaucracy
The purpose is preventing recurrence, and that requires people describing what actually happened - including their own mistakes - without fear. The moment a postmortem produces consequences for individuals, the accounts become defensive and their diagnostic value evaporates.
Keep the format light: timeline, impact, what happened, what made it worse, what made it better, and specific actions with owners. An hour of discussion and a page of notes is enough for most incidents. The discipline that matters is not the document but whether the actions actually get done - postmortem actions that reliably expire teach the team that the exercise is theatre.
- Focus on systemic causes, never individual error.
- Ask what made detection or recovery slow, not just what broke.
- Assign every action an owner and a date.
- Review outstanding actions regularly or stop pretending to create them.
Topics
Marcus Hale
Lead Architect · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.