Peak season failures are rarely surprises in hindsight. The checkout that times out under load was slow in September, the payment provider that rate-limited you published the limit, and the inventory service that oversold had a known race condition somebody filed a ticket about. Peak does not introduce new weaknesses, it removes the slack that was hiding the existing ones. Readiness work is mostly about finding them while there is still time to act. This is core commerce and retail engineering.
01Start in September, because the freeze comes first
Most retailers impose a change freeze in the weeks before peak, which means the real engineering deadline is a month or more before the traffic arrives. A readiness programme that begins in October has already lost the window in which meaningful architectural changes were possible.
Work backwards from the freeze date. Anything requiring a schema change, a provider negotiation or a capacity reservation needs to be finished before it, leaving only configuration and content changes for the peak period itself. Teams that discover this ordering late end up shipping risky changes in the final week, which is how peak outages are manufactured.
02Find the limit before the customers do
Most teams know their platform is fast enough today and have no number for where it stops being fast enough. Readiness requires that number, and getting it means loading the system until it fails rather than until it matches last year's peak.
Test to breaking point in a production-equivalent environment, then identify what broke first. That component is your ceiling, and every other optimisation is irrelevant until it is raised. Repeat until the ceiling is comfortably above the forecast, with genuine headroom rather than a margin that assumes everything else goes right.
- Load to failure, not to the forecast; the forecast is frequently wrong upwards.
- Test the full journey including payment, not just page rendering.
- Include the search and recommendation tiers, which degrade earlier than expected.
- Run against real catalogue size, because query plans change with data volume.
- Record the failure order so you know what to watch on the day.
03Third parties are the part you do not control
A checkout depends on payment processing, fraud scoring, tax calculation, address validation, shipping rates and often a loyalty service. Each has its own capacity and rate limits, and your peak may not be a peak for them, which means their limits apply exactly when you need them not to.
Contact every provider before the season and confirm the limits in writing, along with what happens when you exceed them. Then design for each one being unavailable: a fraud service that times out should not block a sale, and a shipping rate service that fails should fall back to a flat rate rather than emptying the basket.
| Dependency | If it fails or throttles | Acceptable degradation |
|---|---|---|
| Payment provider | Checkout stops entirely | Secondary provider or queued retry |
| Fraud scoring | Orders block | Accept and review after the fact |
| Tax calculation | Totals cannot be computed | Cached rates by region |
| Shipping rates | Basket cannot complete | Flat-rate fallback |
| Recommendations | Pages slow or error | Hide the module entirely |
04Oversell is a business decision, not a bug
Under peak concurrency, strict inventory correctness and high checkout throughput pull against each other. Locking stock for the duration of a checkout is correct and slow; optimistic decrementing is fast and occasionally sells the last item twice.
This is a commercial trade-off and the business should make it explicitly rather than inheriting whichever behaviour the code happens to have. For low-value high-volume goods a small oversell rate handled by an apology and a refund is usually cheaper than the conversion lost to a slow checkout. For limited editions the opposite holds, and the reputational cost of cancelling is severe.
05Decide what to switch off, in advance
Under overload, shedding the features that do not complete a sale preserves the ones that do. This works only if the switches exist before the incident, because writing a feature flag during an outage is not a plan.
Rank everything by its contribution to revenue. Browse, basket, checkout and payment are untouchable. Personalisation, recommendations, live chat, review rendering, wish lists and analytics beacons can all be disabled under pressure with limited commercial impact, and together they often account for a surprising share of the load.
- Build an operations switchboard of flags that disable non-essential modules without a deploy.
- Test each flag in advance so nobody discovers that disabling reviews breaks the product page.
- Write down the order in which they get used, and who is authorised to use them.
- Cache aggressively at the edge for anonymous browsing, which is most of peak traffic.
- Serve a static queue or waiting page rather than letting the whole site time out.
06Rehearse the response, not only the system
On the day, the difference between a bad hour and a bad season is how fast the team notices and how clearly they can act. That depends on rehearsal: a staffed rota, a known escalation path, dashboards showing business metrics rather than only infrastructure ones, and a decision-maker available for the trade-offs that need one.
Watch orders per minute alongside error rates. Infrastructure dashboards can look entirely healthy while conversion has collapsed because of a broken third-party script, and that failure mode is invisible to every technical metric you are likely to be alerting on.
Topics
Marcus Hale
Lead Architect · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.