The outage postmortem usually reads the same way. A third-party API got slow, not unavailable. Requests queued waiting for it. Threads and connections filled up. Requests that had nothing to do with that vendor started failing, because there was nothing left to serve them with. One non-critical dependency took down the entire product, and the vendor never technically went down.
01Slow is worse than down
An endpoint returning connection-refused immediately is easy to handle: you catch it, you degrade, you move on. An endpoint taking thirty seconds to answer is far more dangerous, because every request waiting on it holds a thread, a connection and memory for thirty seconds. Under normal traffic that is enough to exhaust the pool.
This is why timeouts are the first control, not the last. Every outbound call needs one, it needs to be shorter than your own request budget, and it needs to be set explicitly. Most HTTP clients default to no timeout at all, which means the default behaviour is to wait forever.
| Stage | What happens | Time to failure |
|---|---|---|
| Vendor degrades | p99 goes from 200ms to 20s | Immediate |
| Pool saturates | All connections held by waiting calls | Seconds to minutes |
| Unrelated requests queue | Endpoints not using the vendor start timing out | Minutes |
| Health checks fail | Orchestrator restarts healthy instances | Minutes |
| Full outage | Restart storm, cold caches, thundering herd | Minutes |
02What a circuit breaker actually does
A breaker watches the failure rate for one dependency. Below the threshold it stays closed and traffic flows normally. Above it, the breaker opens and calls fail instantly without touching the network. After a cooling period it half-opens, letting a small number of probes through to see whether the dependency has recovered.
The point is not to fix the vendor. It is to fail fast so your own resources stay available. A request that fails in one millisecond returns a thread to the pool; a request that fails in thirty seconds does not. Under a vendor incident that difference is the difference between a degraded feature and a dead product.
03Bulkheads: separate pools per dependency
A breaker alone still lets one dependency consume shared capacity while it degrades. Bulkheads fix that by giving each dependency its own connection pool and concurrency limit, so the noisy neighbour can only exhaust its own allocation.
Sizing is a judgement call. Too small and you throttle a healthy dependency during a legitimate spike; too large and the bulkhead does not contain anything. Start from the concurrency the dependency actually sustains at its p99 latency, add headroom, and revisit it after you have seen a real incident.
| Control | Prevents | Cost of getting it wrong |
|---|---|---|
| Timeout | Unbounded waiting | Too short: false failures on healthy calls |
| Retry with backoff | Transient blips surfacing to users | Too aggressive: you DDoS a recovering vendor |
| Circuit breaker | Sustained failure consuming resources | Too sensitive: flapping open on normal variance |
| Bulkhead | One dependency starving the others | Too small: artificial throttling |
04Decide what degraded looks like before you need it
Opening the breaker raises a question the code has to answer: what now? Serve a cached value, queue the work for later, use a fallback provider, or show an honest error. That decision belongs to the product, not to whoever is on call at the time.
Write it down per dependency. Recommendations service down - render the fallback list. Payment provider down - queue and retry, tell the customer their order is confirmed but payment is pending. Address validation down - accept the address unvalidated and flag it. Each of those is a different answer, and none of them is obvious from the code.
05Testing the thing you hope never runs
Breaker logic that has never opened in a test is not a control, it is a hypothesis. Point the client at a stub that stalls, assert the breaker opens, assert the fallback path executes, assert it closes again when the stub recovers. Run that in CI so the behaviour cannot rot.
Then rehearse it in a staging environment with real traffic shape. Teams that only unit-test their breakers routinely discover in production that the fallback path throws, or that the half-open probe stampedes the recovering vendor because every instance probes simultaneously.
Topics
Priya Iyer
Staff Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer