Why your deploys drop requests: graceful shutdown in Kubernetes

01The race that causes it

When a pod is terminated, two things happen concurrently: the container receives SIGTERM, and endpoint removal begins propagating so load balancers stop routing to it. These are independent processes with no ordering guarantee.

If your application exits promptly on SIGTERM, it can be gone before the propagation completes - and requests routed in that window hit nothing. The application did what it was told; the routing had not caught up.

02The fix is to wait before exiting

The counterintuitive answer is that on SIGTERM your application should keep serving for a short period rather than shutting down immediately. That window lets endpoint removal propagate, so by the time you stop accepting connections nothing is being routed to you.

A preStop hook that sleeps for a few seconds achieves this without application changes, which makes it the pragmatic first fix. The right duration depends on how quickly your ingress and service mesh converge - measure it rather than guessing, but a handful of seconds is typical.

03The shutdown sequence that works

Order matters, and each step exists for a reason. Fail readiness first so nothing new is routed, keep serving for the propagation window, then stop accepting new connections while letting in-flight requests finish, then close dependencies and exit.

04The grace period must exceed your longest request

Kubernetes waits for the termination grace period, then sends SIGKILL regardless of what is happening. If a request takes longer than that period, it is severed mid-flight - and the client sees a connection reset rather than an error it can interpret.

Set the grace period above your longest realistic request or background job, including the propagation wait at the start. Long-running jobs need particular attention: they either need to complete within the window or be designed to resume, and discovering which during an incident is unpleasant.

• Grace period > propagation wait + longest request duration.
• Background workers need resumability or bounded runtime.
• Verify the process actually exits before SIGKILL fires.
• Log shutdown steps so you can confirm ordering in production.

05Signals that do not reach your process

A container whose entrypoint is a shell script often has that shell as process one, and it may not forward SIGTERM to the application. The application then never begins shutting down and is killed at the end of the grace period every time.

The symptom is that shutdown always takes exactly the grace period. If deploys consistently take that long, check what process one actually is - using exec form for the entrypoint or a minimal init process usually resolves it immediately.

06Verifying it works

Run a steady stream of requests during a rolling deploy and count non-200 responses. This is the only test that matters, because reading the configuration cannot tell you whether propagation timing on your cluster is what you assumed.

Do it after any change to ingress, service mesh or shutdown handling. The window is timing-dependent, so a configuration that works today can start dropping requests after an unrelated infrastructure change.