Autoscaling that responds too late: metrics, warm-up time and the spike you will miss

01Scaling latency is a sum, not a setting

Between load arriving and capacity serving it there are five distinct delays: the metric collection interval, the evaluation or stabilisation window, the scheduling decision, node provisioning if no space exists, and application warm-up before the instance is genuinely useful. Each is configured in a different place, often by different people.

Add them honestly and the total is regularly three to six minutes on Kubernetes with cluster autoscaling in play, and longer for JVM services that need class loading and JIT warm-up before they hit steady-state throughput. A spike that peaks in ninety seconds is over before your first new pod is ready.

02CPU is a lagging indicator of the wrong thing

Average CPU tells you what the fleet has already absorbed, not what is queuing. For an I/O-bound service it may barely move while latency triples, because the threads are blocked rather than busy. Scaling on it means reacting after the backlog exists.

Scale on the signal closest to the work waiting to be done. For a queue consumer that is queue depth, or better, oldest-message age, which directly expresses the SLO you care about. For a request service, in-flight concurrency per instance is a far earlier signal than CPU. Request rate works well when the cost per request is uniform, and badly when it is not.

• Queue workers: scale on oldest-message age, not consumer CPU.
• Request services: scale on concurrency or p95 latency, not average CPU.
• Mixed workloads: split into separate deployments before choosing a metric.
• Whatever you choose, verify it moves before user-visible latency does.

03Scale up fast, scale down slowly

The costs are asymmetric and the configuration should be too. Scaling up too eagerly costs you some money for a few minutes. Scaling down too eagerly costs you an outage when the load returns thirty seconds later and you are back at the bottom of the provisioning curve.

Configure aggressive scale-up - short stabilisation, large step size, permission to double - and conservative scale-down with a stabilisation window of several minutes and small decrements. Thrashing is the classic symptom of symmetric settings: the fleet oscillates, every cycle pays warm-up cost, and average latency is worse than it would have been with no autoscaling at all.

04Warm-up is what breaks the illusion

A new instance passing its readiness probe is not the same as a new instance carrying its share of traffic. Connection pools are empty, caches are cold, the JIT has not compiled the hot path, and in some runtimes the first few hundred requests are several times slower than steady state.

Load balancers that route by round-robin will send that instance a full share immediately, and its latency drags the p99 for the whole service - so scaling up briefly makes the metrics worse, which some controllers then respond to by scaling up further. Fix it at the source: make the readiness probe exercise the real dependencies, pre-warm connection pools during startup, and use slow-start or least-outstanding-requests routing so new instances ramp rather than being thrown in.

05Predictable load should be scheduled, not discovered

Most traffic is not a surprise. A weekday morning ramp, a batch window, a campaign send at a known time, a market open - these repeat and are visible in a month of history. Waiting for reactive autoscaling to rediscover them every day guarantees you are late every day.

Use scheduled scaling for the known shape and reactive scaling for the deviation on top of it. Set the floor high enough to absorb the fastest realistic ramp, and keep headroom sized against your measured scaling latency rather than a round number. Headroom of twenty to thirty percent sounds wasteful until you price it against the alternative of a five-minute degradation during your highest-value hour.

• Schedule the recurring shape; react only to the residual.
• Set minimum replicas from the traffic floor, not from cost anxiety.
• Size headroom against measured scale-up time, not a guess.
• Keep spare node capacity with low-priority placeholder pods to skip provisioning.

06Autoscaling does not fix a saturated dependency

Adding instances multiplies the load on everything downstream. Scale from ten to forty pods and your database sees four times the connections, your third-party API hits its rate limit, and your cache stampedes. The service scales beautifully and the system fails anyway - now with more moving parts.

Before raising a maximum, work out what that maximum means for every shared dependency. Cap connection pools so total connections at max replicas stay under the database limit. Rate-limit outbound calls in aggregate rather than per instance. And keep load shedding in place regardless, because autoscaling has a floor on its response time and the only thing that protects the system inside that window is refusing work gracefully.