Timeouts and retries: the cascading failure you are configuring by default

01The default timeout is usually no timeout

Most HTTP clients, database drivers and RPC libraries ship with either no timeout or one measured in minutes. That default is chosen so that nothing appears broken during development, and it is the single most dangerous configuration in a distributed system, because a call with no timeout converts a slow dependency into an indefinitely held resource.

The resource is what matters. Each waiting call holds a thread or connection, and the pool is finite. At 200 requests per second against a dependency that slowed to 3 seconds, you need 600 concurrent slots where 8 sufficed before. Your pool has 50. The other 550 requests queue, the queue exceeds the caller's own timeout, and a service that is entirely healthy starts failing requests that never needed the slow dependency at all.

02Timeouts must decrease down the call chain

The common misconfiguration is uniform timeouts: every service sets 30 seconds because it seemed reasonable in isolation. Now a user-facing request with four hops can legitimately take two minutes, and the caller at the top has given up long before the work at the bottom finishes - so the work continues, consuming resources, producing a result nobody will read.

Allocate a budget from the outside in. The edge gets the user-facing SLO, and each hop subtracts its own processing time and passes the remainder downstream as an explicit deadline. A service that receives a request with 200 milliseconds remaining and knows its dependency typically takes 300 should fail immediately rather than start work it cannot finish. gRPC propagates deadlines natively; over HTTP you pass a remaining-milliseconds header and honour it, which is a small amount of code and disproportionately valuable.

03Retries amplify load precisely when you can least afford it

Retries look like resilience and behave like a load multiplier. Three attempts per call means a struggling dependency receives three times the traffic at the exact moment it is failing, and it will not recover under three times the load - it will get slower, causing more timeouts, causing more retries.

The multiplication compounds across layers. If four services in a chain each retry three times, a single user request can produce 81 calls at the bottom. Retries must be configured at exactly one layer of the chain - typically the outermost one that can make a meaningful decision - and every layer below it must be explicitly configured not to retry. Left-alone client library defaults are where the hidden multipliers live.

• Retry at one layer only; disable retries explicitly everywhere else.
• Never retry non-idempotent operations without an idempotency key.
• Never retry a 4xx - the request will fail identically every time.
• Cap total attempts and respect the remaining deadline, not just the attempt count.

04Backoff without jitter creates a synchronised stampede

Exponential backoff - wait 100ms, then 200, then 400 - is standard and insufficient on its own. When a dependency has a brief outage, every client fails at approximately the same moment and therefore retries at approximately the same moment. The dependency comes back, receives the entire fleet's retries in a single burst, falls over, and the cycle repeats with a longer period.

Jitter breaks the synchronisation, and full jitter - a random wait uniformly distributed between zero and the current backoff ceiling - is the variant that performs best in practice. It spreads the retry load across the interval instead of concentrating it at the boundary, and it costs one line of code. Decorrelated jitter, where each wait is derived randomly from the previous one, performs similarly well and avoids a hard ceiling.

05Retry budgets beat retry counts

Per-call retry limits still permit unbounded aggregate amplification: if every call is failing, every call retries, and your fleet triples its outbound traffic. A retry budget caps retries as a proportion of total requests - allow retries only while they remain under, say, 10% of successful traffic - so isolated failures retry normally and systemic failures do not multiply.

This is a small piece of shared client-side state, and it converts retry policy from something that is safe in the common case to something that is safe in the failure case too. It is what service meshes implement when they expose a retry budget, and it is worth implementing in a shared HTTP client if you do not have a mesh. Expose the budget-exhausted counter as a metric; it is an excellent early signal that a dependency is degrading.

06Circuit breakers stop you waiting for a known-dead dependency

When a dependency is reliably failing, the fastest correct action is not to call it. A circuit breaker tracks the recent failure rate and, past a threshold, opens - rejecting calls immediately without consuming a connection or a thread. After a cooling period it half-opens, allowing a small number of trial requests, and closes again if they succeed.

The value is resource preservation on the caller's side: an open breaker means your threads are free to serve requests that do not need the broken dependency, which is exactly what prevents the cascade. Trip on rate over a rolling window rather than on consecutive failures, and require a minimum request volume, or a service handling two requests a minute will trip the breaker on a single transient error.

• Trip on failure rate within a window, with a minimum-volume threshold.
• Include timeouts in the failure rate - slow is the failure you care about.
• Allow only a small number of trial requests when half-open.
• Define the fallback: cached data, a degraded response, or a clear error - never a silent empty result.

07Bulkheads: one dependency should not consume every thread

Even with timeouts and breakers, a shared connection pool means a slow dependency can occupy every slot before anything trips. Bulkheading gives each downstream dependency its own bounded pool, so the slow one exhausts its own 20 connections and the other dependencies keep their capacity. Calls to the saturated dependency fail fast at the pool boundary rather than queueing.

Size each bulkhead from measured concurrency - arrival rate times normal latency, with headroom - rather than from a round number. This is also the point at which you decide which dependencies are critical: a request that can complete without the recommendations service should not fail because the recommendations pool is full, and expressing that as a bulkhead plus a fallback makes the degradation deliberate rather than accidental.

08Test it, because none of this is verifiable by reading configuration

Every mechanism here is only real if it has been exercised. Inject latency into a dependency in a staging environment under representative load and watch what happens: whether the timeout fires at the value you configured, whether the breaker opens, whether the caller degrades or dies. The number in the config file and the behaviour under load diverge more often than anyone expects, usually because a library default overrides it somewhere in the stack.

Then measure the things that predict the failure rather than describe it. Pool utilisation and queue wait time rise well before error rates do, so they give you warning; retry rate as a fraction of total requests tells you a dependency is degrading before it fails outright. Alert on those leading indicators, and the cascading failure becomes a page at 15% saturation rather than an outage at 100%.