![\"Two](\"/77a080e6e9f9496080ace41f97f53229/requests-vs-limits.svg\")
A service that ran fine all week starts dying. kubectl get pods shows it restarting, and kubectl describe has one part that matters:
State: Running\n Last State: Terminated\n Reason: OOMKilled\n Exit Code: 137Nothing in the code changed, and the image is the same one that shipped on Monday. What changed is how much memory the container needed. It crossed a line you set weeks ago and forgot about.
\nThis post is for engineers who run services on Kubernetes and have hit OOMKilled without a clear picture of what is going on. It covers what memory requests and limits actually do, why the two kills that mention memory are different events, and which settings actually stop the kills. I ran the Kubernetes side of the CMS workflow operations stack at CERN. There, a WMCore service quietly growing past its memory limit is the difference between a slow afternoon and a paged night.
Start by reading the exit code correctly. 137 is 128 + 9. The 128 + is the shell convention for “terminated by a signal”, and 9 is SIGKILL.
So the container did not throw an exception, did not log a stack trace, and did not get a chance to clean up. Something outside the process sent it a SIGKILL, which a process cannot catch or handle, and the process stopped mid-instruction. Pair that with Reason: OOMKilled and you know who pulled the trigger: the Linux kernel’s out-of-memory (OOM) killer. It acted because the container’s cgroup (the kernel mechanism that caps a group of processes’ resources) hit its memory ceiling.
That detail rules things out. A SIGKILL from the OOM killer means no graceful shutdown ran, so any “on exit, flush the buffer” logic did not fire. If you are debugging lost writes alongside an OOMKill, that is why. The process did not get to say goodbye.
CPU and memory look like the same kind of resource in a pod spec. Under pressure they behave nothing alike.
\nCPU is compressible. When a container wants more CPU than its limit, the kernel throttles it. The container runs slower, and everyone survives.
\nMemory is incompressible. You cannot run a process on “less memory than it is asking for right now.” Once a container’s allocations reach its memory limit, there is nothing to throttle. The kernel’s only lever is to kill something in the cgroup and reclaim the pages.
\nSo the mental model is simple: a CPU limit slows you down; a memory limit kills you. Every OOMKill follows from that asymmetry.
\nThe pod spec asks for two memory numbers per container. Two different parts of Kubernetes read them, at two different times. The comments in this snippet say which part reads which:
\nresources:\n requests:\n memory: \"512Mi\" # what the SCHEDULER reserves\n limits:\n memory: \"1Gi\" # what the KERNEL enforcesThe request is a scheduling input. The kube-scheduler places your pod on a node only if the node’s existing memory requests plus yours still fit in the node’s allocatable memory. The request is a reservation. Once the pod is scheduled, the request has mostly done its job, and the running container is not held to it.
\nThe limit is a runtime input. It becomes the memory ceiling on the container’s cgroup, and the kernel enforces it directly. Cross it and the OOM killer runs.
\nThe scheduler never looks at limits when it places a pod. That causes the most dangerous surprise: limits can add up to more than the node has. The scheduler packed the node by requests and cheerfully let the limits overcommit it.
\n
Keep the gap between those two bars in mind, because both failure modes live there:
\nWhen people say “the pod got OOMKilled,” they usually mean one specific thing. But Kubernetes has two distinct memory kills, and if you confuse them you end up debugging the wrong layer.
\n![\"Two](\"/01a4f7b116c1b6e850683c952f623c97/memory-kill-triggers.svg\")
Container-limit OOMKill. One container reaches its own limit. The cgroup OOM killer terminates the process in that container, the kubelet reports Reason: OOMKilled, and the pod restarts according to its restartPolicy. This kill is local. The node might have gigabytes free; this one container simply asked for more than its own cap allowed. This is the kill you see in kubectl describe pod.
Node-pressure eviction. The node runs low on memory, whatever any single container’s limit is. The kubelet (the agent on each node that runs pods) monitors node memory. When free memory drops below its eviction threshold, it proactively evicts whole pods to reclaim memory. This shows up as Reason: Evicted, not OOMKilled, often with a message like The node was low on resource: memory. The evicted pod may have been well within its own limit. It was chosen because the node as a whole was in trouble.
To tell which one you hit, check the reason field and where it appears:
\nOOMKilled on the container in describe pod means a limit problem for that container.Evicted on the pod, plus MemoryPressure on the node in kubectl describe node, means a node-packing problem.The two have different fixes, so read the field before you change anything.
\nWhen the kubelet evicts pods under node pressure, it does not choose at random. It ranks pods by their Quality of Service class (QoS class). Kubernetes assigns the class based on how you set requests and limits:
\nIn practice, this means the request is not only a scheduling number; it also sets your place in the eviction order. A pod running under its request is a poor eviction candidate. A pod running well over its request is the first Burstable pod to go. A request that reflects real steady-state usage keeps a healthy pod from becoming collateral damage when a noisy neighbor fills the node.
\nStart with the pod, then widen out. The exit reason tells you which of the two kills you are dealing with. These three commands show the last exit reason per container, the full pod status and events, and whether the node itself is under memory pressure:
\n# Which containers were OOMKilled, and their last exit reason\nkubectl get pod <pod> -o jsonpath='{range .status.containerStatuses[*]}{.name}{\": \"}{.lastState.terminated.reason}{\" (\"}{.lastState.terminated.exitCode}{\")\"}{\"\\n\"}{end}'\n\n# Full picture: events, restart count, the Terminated block\nkubectl describe pod <pod>\n\n# Is the NODE under memory pressure? (eviction vs container OOM)\nkubectl describe node <node> | grep -A5 ConditionsIf it is a container OOMKill, the next question is how close the container runs to its limit in normal operation. Compare the limit against actual usage, not against the request. If you have metrics-server or Prometheus, the working set is the number to watch:
\nkubectl top pod <pod> --containersThat command gives you a single reading. Look at container_memory_working_set_bytes over time instead. The working set is roughly the memory the kernel cannot reclaim under pressure, and that is what the OOM decision is made against.
For example, a container that idles at 300Mi and spikes to 1.1Gi on a large request needs a limit above that spike. A single snapshot from kubectl top will not show you the spike. Graph it.
There is no universal right value, but there is a reliable procedure.
\ncontainer_memory_working_set_bytes. You want the p99 of that curve, not the average, because the OOM killer does not care about your average.This is what step 4 looks like in the spec:
\nresources:\n requests:\n memory: \"1Gi\"\n limits:\n memory: \"1Gi\" # request == limit -> Guaranteed QoSThe runtime does not see the cgroup limit. This is the classic one. A JVM from before container awareness sizes its heap from the host’s total memory, not the container limit. It happily grows a heap far past the cgroup cap, then gets OOMKilled while it still thinks it has plenty of “free” heap. Runtimes handle this differently:
\nUseContainerSupport (on by default) plus -XX:MaxRAMPercentage.GOMEMLIMIT for the same reason.If your OOMKilled container hosts a VM-based runtime, check that the runtime knows about the limit before you touch the limit itself.
\nCrashLoopBackOff that is really a repeating OOMKill. A container that OOMKills on startup, restarts, and OOMKills again lands in CrashLoopBackOff. The crash-loop label hides the memory cause. Always read lastState.terminated.reason. If it says OOMKilled, the fix is memory, not the backoff timing or the readiness probe.
Blaming the limit when the node was the problem. If the reason is Evicted and the node shows MemoryPressure, raising the pod’s limit does nothing, because the pod was not over its limit. The node was overcommitted. Fix it at the packing layer: reduce overcommit by setting honest requests across the workloads on that node, or add capacity. I covered the safe-rollout side of this in ArgoCD sync waves. Getting requests right keeps a GitOps deploy from quietly overpacking a node over successive releases.
A memory leak disguised as a bad limit. If a container’s working set climbs steadily and never levels off, raising the limit only spaces out the same OOMKills. A leak needs fixing, not a bigger cap. The tell is the shape of the curve: a genuine peak-and-return workload plateaus, while a leak trends up forever. I push exactly this into the dashboards described in OpenSearch dashboards for workflow monitoring. “Is this a spike or a leak?” is a question a graph answers in seconds and kubectl describe never will.
A liveness probe restarting an OOM victim into a loop. If a container is being OOMKilled and also has an aggressive liveness probe, you can lose track of which one restarted it. Keep the two concerns separate; I wrote about not overloading health checks in Kubernetes probes. An OOMKill is the kubelet reacting to the kernel, not to a failed probe, and the exit reason tells you which happened.
\nEarly on, I set memory limits by copying whatever number the last service used. That is how you end up with a fleet of limits unrelated to what anything actually needs. Some were far too tight and OOMKilled under normal peaks. Others were so loose that a leak ran for hours before anyone noticed.
\nThe habit that fixed it was boring:
\nRequests and limits stopped being magic numbers and became two measurements with two jobs.
\nThe other change was treating a restart with OOMKilled as a first-class alert, not something I noticed while debugging something else. A container that OOMKills is telling you its real memory need has outgrown the box you drew around it. You want to know that on the day it starts, not in the week the restarts get bad enough to page someone.
This is the memory discipline behind the WMCore and Unified operations services. They schedule Monte Carlo production and reconstruction for the CMS experiment across the Worldwide LHC Computing Grid (WLCG). Those services process bursty workloads of varying size, so the gap between steady state and peak is real and worth measuring rather than guessing.
\nGetting requests and limits right is not glamorous work. But it decides whether a node packs cleanly and degrades gracefully, or instead wastes half its memory on unused reservations or falls over the first time three pods spike at once. The kernel’s OOM killer is not the problem. It is doing exactly what an expectation you never properly set told it to do.
\nDiagrams by M. Hassan Ahmed, made for this post.
\nImage credit: Diagrams created by M. Hassan Ahmed for this post, released under CC0 (public domain).
","frontmatter":{"title":"Kubernetes OOMKilled: Requests vs Limits","date":"2026-07-15T00:00:00.000Z","description":"Your pod died with OOMKilled and exit 137. What Kubernetes memory requests and limits actually do, what triggers the kill, and how to stop it happening.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAACJklEQVQozyVRy27TQBT1CvKo52mPZ8avafyIHduJ4zgpdVxaSlsEAipaBAiExJI9S36ADbsKVPGzjEG6ujq6M0fnnnsM4fW6mOh8ddZs366am3pzU9XX6/Z2qfvmdlG9rJs3dTMMpX/iyE64/f8yxkiOoQB26AR5lLdpdZSvuqTc6q7xoumjYpvX+/mqixZb288BOxwBMYZyglzDpDNgJ8iZQzYnIoMsndLItOIDGk1xCHGIcAhQYLGYOYllR9SaYaIACfWrAZ2UeEssSyyXZXulFidIlMStsFtCqjgJHRIyHCwXj453Tz2eaqyHFAcT5BuAJcRdQp5jN3VUi2Q2ZTGWFZYFIGpgEkWRI/mMWxEyLQo9hyqCgzH0NDlFsqDuet/9uDz/83h/v9l9scMSCU3WsopAVqUfnp/9Pj++uzj6HquKIp8STXaNwbDMz/pfTfMNiiovXnx89cmbVYAXUGtCu0zfX/X3oagxSV9vP989u7aQp5UnQBoT4sbZu777+YAIHpTzNGNeVhQt/KdsIX7R3Xl8jYAdsFlse183J3M5N6E7BsIwLcVVz8M9cRe7ts3LDeAlDwfPJhrsKbdjJFh52WWySpwYYyWsQ4zckcn12rE+LBD5LD/eHJ2ud6d2uEayQrIEyB8Oi0TqZk+Suo9WjSo5Ha5IkDcymTFFvk4V2JE2P9HZWjFkiWnNTHo4NTmGUv+zSYCwBzWgOiRPy5pAPDxgfwFwmlj+1mWLqQAAAABJRU5ErkJggg==","aspectRatio":1.899441340782123,"src":"/static/6399392464251e994ff9f039b0376535/40a76/hero.png","srcSet":"/static/6399392464251e994ff9f039b0376535/c972b/hero.png 340w,\n/static/6399392464251e994ff9f039b0376535/27625/hero.png 680w,\n/static/6399392464251e994ff9f039b0376535/40a76/hero.png 1360w,\n/static/6399392464251e994ff9f039b0376535/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-07-15-kubernetes-oomkilled-requests-vs-limits/","previous":"blog/2026-07-16-cancel-llm-stream-react-abortcontroller/","next":"blog/2026-07-19-indirect-prompt-injection-rag/"}},"staticQueryHashes":["32046230"]}