Pods restart every few hours with OOMKilled, yet the memory dashboard shows usage comfortably below the limit. Both observations are accurate. Dashboards typically display averages over a sampling interval, and the kernel reacts to instantaneous usage - a spike lasting two seconds can trigger termination while never appearing on a graph averaged over a minute.
01Requests and limits do different jobs
The request is what the scheduler reserves when placing the pod. The limit is the ceiling the kernel enforces. Setting them far apart means pods are scheduled onto nodes based on a small reservation and then permitted to grow into memory that may not be available, producing evictions and terminations under pressure.
Memory limits are enforced by termination rather than throttling, unlike CPU. There is no graceful degradation: exceeding the limit by a small amount ends the process immediately, which is why exit code 137 arrives with no application error preceding it.
| Setting | Effect | When wrong |
|---|---|---|
| Memory request too low | Scheduled onto a crowded node | Eviction under node pressure |
| Memory limit too low | Killed at the ceiling | Repeated OOMKilled restarts |
| No limit set | Can consume the node | Neighbouring pods evicted |
| Request equals limit | Guaranteed class, predictable | Usually the safest default |
02Runtimes size themselves from the wrong number
Older runtimes inspect the host machine rather than the container's cgroup when deciding how much memory to use. A container limited to five hundred megabytes on a sixty-four gigabyte node can have a runtime that believes it may allocate many gigabytes, so it delays garbage collection until well past the container ceiling and is killed before it ever collects.
Set the runtime's heap ceiling explicitly, below the container limit, leaving room for non-heap memory - thread stacks, buffers, native allocations and the runtime itself. A common starting point is around three quarters of the container limit for the heap.
03Find the spike, not the average
Increase the resolution of memory metrics to catch short peaks. Container memory working set at a fifteen second interval reveals spikes that a one minute average smooths away entirely, and the spike is what triggered the kill.
Correlate the restart timestamps with application activity. Restarts clustered at a specific time typically indicate a scheduled job, a large report, a bulk import or a periodic cache refresh loading more into memory than steady-state operation requires.
| Cause | Pattern |
|---|---|
| Loading a full result set into memory | Spikes proportional to tenant size |
| File upload or export buffered entirely | Spikes with file size |
| Scheduled batch job | Regular, clock-aligned |
| Cache warm on startup | Spike immediately after deploy |
| Slow leak | Sawtooth rising to the limit over hours |
| Concurrency burst | Spikes with traffic peaks |
04A sawtooth means a leak, not a limit problem
Memory that climbs steadily to the ceiling, gets killed, restarts and climbs again at the same rate is a leak. Raising the limit only extends the interval between restarts and postpones diagnosis, usually to a point where the pattern is less obvious.
Capture a heap snapshot before the process is killed to identify what accumulates. Frequent culprits are unbounded in-memory caches without eviction, event listeners registered per request and never removed, and collections that grow with request count rather than being scoped to a request.
05Stream instead of buffering
Most memory spikes come from holding an entire dataset in memory when the work could be done incrementally. Reading a million rows to transform and write them requires only a small buffer if streamed, and gigabytes if collected first.
This is the durable fix. Raising limits accommodates today's largest input; streaming accommodates whatever arrives next year, which matters because the record that breaks it is usually from your largest and most important customer.
Topics
Aarav Patel
Principal Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer