Designing for failure: timeouts, retries, circuit breakers and the ways they backfire

01No timeout means an unbounded timeout

The default timeout in many HTTP clients is none at all. A request to a hung service waits indefinitely, holding a connection, a thread and whatever resources the caller allocated. Enough of those and the caller exhausts its own pool and stops serving traffic - having been taken down by a dependency that never returned an error.

Set explicit timeouts on every network call, derived from measured latency rather than round numbers. A p99 of 200ms suggests a timeout around 500ms, not thirty seconds. The purpose is to fail fast enough that resources are released while the system can still recover.

• Explicit connect and read timeouts on every outbound call.
• Derive values from measured p99, not convenient round numbers.
• Ensure the caller's total budget exceeds the sum of downstream timeouts.
• Timeouts must shrink as you go deeper in the call chain.

02How retries cause outages

A service degrades and starts failing 20% of requests. Every caller retries three times. Effective load triples against a service already struggling, pushing failure rate higher, causing more retries. The service that would have recovered from a transient blip is now saturated and fully down. This is a retry storm and it is one of the most common self-inflicted outages in distributed systems.

The mitigations are well established and frequently skipped. Exponential backoff spreads retries over time. Jitter prevents synchronised clients retrying in lockstep. Retry budgets cap the proportion of traffic that may be retries - typically around ten percent - so retries can never multiply load without bound.

03Retry only what is safe to retry

Retrying a read is harmless. Retrying a payment may charge twice. Before adding retry logic, establish whether the operation is idempotent, and if it is not, make it so with an idempotency key that the server records and deduplicates against.

Equally important is retrying only the right failures. Connection errors, timeouts and 503s are worth retrying. A 400 will fail identically every time, and a 401 will not fix itself. Retrying deterministic failures wastes capacity and delays surfacing the real error.

04Circuit breakers stop the bleeding

When a dependency is genuinely down, continuing to send requests helps nobody. Each one consumes caller resources waiting for a timeout, and adds load to a service trying to recover. A circuit breaker tracks failure rate and, past a threshold, stops attempting calls entirely for a cooling period, failing immediately instead.

The subtle part is recovery. After the cooling period, allow a small number of trial requests through rather than resuming full traffic. If they succeed, close the breaker gradually. Slamming full production load onto a service that has just come back is a reliable way to knock it down again, and this half-open behaviour is what prevents the oscillation.

05Degrade rather than fail

The best outcome is not a fast error but a partial success. If recommendations are unavailable, show the product page without them. If the personalisation service is down, serve the generic experience. Most applications have a core function and a set of enhancements, and the enhancements failing should never take out the core.

This requires deciding in advance what is essential and what is optional, and building the fallback path deliberately. It also means the fallback must be exercised - a degraded mode that has never run in production is as untrustworthy as an untested failover, and it usually breaks the first time it is genuinely needed.

• Classify every dependency as essential or enhancing before an incident.
• Build and test the degraded path, including under load.
• Cache last-known-good responses for enhancing services.
• Make degradation visible in monitoring - silent fallbacks hide real problems.

06Load shedding beats collapsing

When demand exceeds capacity, a system that accepts everything serves everything slowly and eventually fails at all of it. A system that rejects excess load quickly keeps serving what it accepted. Rejecting 20% of requests immediately is a far better outcome than serving 100% at thirty-second latency and then falling over.

Shed at the edge, before requests consume resources, and shed by priority where you can distinguish. Health checks and existing sessions should survive; speculative background work is the first to go. This is unintuitive to product stakeholders and worth explaining before the incident rather than during it.