The load test sustained twice the expected traffic with latency well within target. Production hit that number and degraded badly. The test was not wrong, it was unrepresentative - and the specific ways it differs from reality are predictable enough to correct for.
01Real traffic is bursty and correlated
Load generators produce evenly spaced requests. Real users arrive in bursts driven by notifications, campaigns, business hours and each other's behaviour. A system comfortable with a steady thousand requests per second can fail at an average of four hundred if those arrive as short intense spikes.
Correlation matters as much as volume. Many clients retrying simultaneously after a brief failure produce a synchronised surge precisely when the system is least able to absorb it, which is how a small blip becomes an outage.
| Test assumption | Production reality | Consequence |
|---|---|---|
| Even request spacing | Bursts and spikes | Queues fill instantly |
| Warm caches | Cold and evicted caches | Origin sees far more load |
| Uniform data access | Hot keys and popular records | Contention on a few rows |
| Healthy dependencies | Dependencies degrading together | Cascading timeouts |
| Fresh connections | Pool near its limit | Waiting on connections, not work |
02Percentiles, not averages
An average latency of eighty milliseconds is compatible with a substantial fraction of users waiting several seconds. Averages conceal exactly the behaviour that generates complaints, and a system is generally judged by its worst common experience rather than its typical one.
Report the ninety-fifth and ninety-ninth percentiles, and remember that a page composed of ten API calls will hit the ninety-ninth percentile of at least one call more often than not. Tail latency compounds across a request graph in a way median latency does not.
03Pools and limits produce cliffs
Connection pools, thread pools and file descriptor limits do not degrade gracefully. Below the limit, latency is flat and everything looks healthy. At the limit, requests begin waiting for a connection rather than for work, and latency rises almost vertically.
This is why the graph shows a cliff rather than a slope, and why a test that stops just short of the limit reports success. Test past the breaking point deliberately - the useful information is where it breaks and how it behaves afterwards.
| Limit | Symptom when reached |
|---|---|
| Database connection pool | Requests queue waiting for a connection |
| HTTP client connections | Outbound calls stall under concurrency |
| Thread or worker pool | Throughput plateaus, latency climbs |
| File descriptors | Connection refused errors under load |
| Downstream rate limit | Sudden burst of 429 responses |
04Cold caches change the shape of everything
A test that runs for twenty minutes against the same data develops a cache hit ratio that production never achieves. Production has evictions, deployments that restart processes and empty in-memory caches, and long-tail requests that were never cached at all.
Test with a cold cache at least once, and model what happens if the cache becomes unavailable entirely. Many systems are sized on the assumption of a high hit ratio and cannot survive falling back to the origin for everything, which turns a cache outage into a total outage.
05Test the failure modes, not just the throughput
The valuable outcome of a load test is not a maximum requests-per-second figure. It is knowing what breaks first, what the symptom looks like, and whether the system recovers on its own once load subsides or stays broken until someone intervenes.
Add a dependency slowdown to the test - make a downstream service take two seconds instead of fifty milliseconds - and watch whether timeouts, retries and circuit breakers contain it or amplify it. That single scenario predicts real incidents better than any throughput number.
Topics
Marcus Hale
Lead Architect · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer