![\"At](\"/d7524438167843ff4f9dbe454fa08675/attention-recompute.svg\")
The first time it happens, it makes no sense. A 7B model that served fine all week starts throwing CUDA out of memory in the middle of a request, not at load. The weights fit. They fit yesterday, and nothing about the model changed. What changed is the request: someone pasted a long document, or an agent got twelve tool calls deep into a task, and the request that broke was longer than the ones before it.
The thing eating your VRAM (the GPU’s own memory) is not the weights. It is the KV cache, and unlike the weights it is not a fixed cost. It grows with every token in the conversation, and it grows again with every request you batch together.
\nThis post is for engineers who serve or run LLMs and want to know exactly what that memory is, why it exists, and which levers actually shrink it. I’ve run into it in two places. One is the long-context side of Archi, the RAG copilot for CMS operations, where a single answer can pull in thousands of tokens of logs and docs. The other is agent loops, where the transcript only ever gets longer.
\nA transformer generates text one token at a time. To produce the next token, self-attention compares the current token against every token before it. That comparison runs through three projections of the sequence: queries, keys, and values (Vaswani et al., “Attention Is All You Need”). Roughly, the current token’s query is matched against the earlier tokens’ keys, and the best matches’ values are blended into the output.
\nHere is the part that matters for memory. The keys and values for tokens 1 through N-1 do not change when you generate token N, because they were computed from tokens that are already fixed. Recomputing them at every step is pure waste. Instead, you compute each token’s key and value once, keep them, and reuse them at every later step. That store is the KV cache.
\n
The trade is explicit:
\nYou stop repeating work. In return, you have to hold every past key and value in GPU memory for the whole life of the request. That held memory is the bill.
\nThe cache size is not mysterious. For a single token it is:
\nbytes/token = 2 x num_layers x num_kv_heads x head_dim x dtype_bytesThe 2 is there because you store both keys and values. For plain multi-head attention, num_kv_heads x head_dim is usually just the model’s hidden size (the width of each token’s vector), so a cleaner form is:
bytes/token = 2 x num_layers x hidden_size x dtype_bytesNow put in real numbers. Llama-2-7B has 32 layers and a hidden size of 4096, and in fp16 (16-bit floating point) each value takes 2 bytes:
\nnum_layers = 32\nhidden_size = 4096\ndtype_bytes = 2 # fp16\n\nper_token = 2 * num_layers * hidden_size * dtype_bytes\n# = 2 * 32 * 4096 * 2 = 524,288 bytes -> 512 KB per tokenThat is half a megabyte per token. It looks small until you multiply it by a context length and a batch size, and both of those are linear multipliers. Here is a batch of 8 sequences of 4096 tokens each:
\nseq_len = 4096\nbatch = 8\n\ntotal = per_token * seq_len * batch\n# 512 KB * 4096 * 8 = 16 GBThat is sixteen gigabytes of cache, on top of roughly 14 GB of fp16 weights. On a 24 GB card that request does not fit, and the failure lands at generation time, not at startup. The chart shows the cache stacking on top of the fixed weights as the batch grows.
\n![\"On](\"/21ac5394a875b800bc07a1558409c16e/kv-cache-growth.svg\")
Two things follow from this:
\nKnowing the math for one sequence is not enough. A real serving engine runs many sequences at once, each with a different, unpredictable length. If you reserve a contiguous block big enough for the maximum context of every request, most of that reservation sits empty for the requests that finish early or never grow that long.
\nThe vLLM team measured this and found that older serving systems wasted 60–80% of KV cache memory to fragmentation and over-reservation (Kwon et al., “Efficient Memory Management for LLM Serving with PagedAttention,” SOSP 2023). Their fix, PagedAttention, borrows the idea of paged virtual memory from operating systems. It stores the cache in small fixed-size blocks that do not have to be contiguous, and hands them out on demand. That drops the waste to under 4%. Because the freed memory becomes a bigger batch, it also raises throughput substantially (vLLM launch post). If you serve models and are not on an engine that pages the cache, this is usually the single biggest win available to you.
\nEvery term in 2 x num_layers x num_kv_heads x head_dim x dtype_bytes is a place to cut. The useful levers each attack a different term, and they stack.
![\"Four](\"/54d1add268d4f06be366fc458348dcbb/shrink-kv.svg\")
Grouped-query attention (GQA) cuts num_kv_heads. Instead of every query head owning its own keys and values, groups of query heads share one set. Llama-2-70B uses 64 query heads but only 8 KV heads, an 8x reduction in cache size, and the GQA paper reports under 0.5 perplexity loss at that ratio. GQA is a property of the model, not something you turn on at serving time, so it is a reason to prefer a GQA model when memory is tight. Most current open models (Llama 3, Mistral, Qwen) already use it.
Quantizing the cache cuts dtype_bytes. Storing keys and values in int8 instead of fp16 halves the cache, for a modest and usually acceptable quality cost. Serving engines expose this as a flag. It is independent of GQA, so you get both factors.
Paging cuts waste rather than the theoretical size, as described above. For a busy multi-tenant service it is usually the biggest single win.
\nShorter context cuts seq_len, and it is the one lever that lives in your code rather than in the model or the engine. In a RAG system, it means retrieving fewer, tighter chunks instead of stuffing the window. (I wrote about measuring retrieval quality and reranking partly for this reason: better ranking lets you send less.) In an agent loop, it means summarising or trimming old turns instead of replaying the entire transcript on every step. The cache grows linearly with what you put in the window, so discipline in the window is discipline in VRAM.
If I were sizing a deployment again, I would start from the variable cost, not the weights. Decide the longest context you will actually admit, multiply by the batch size you need for throughput, and check that weights + KV fits with headroom before picking hardware. That single calculation catches most surprises.
The order of levers I reach for:
\nThe first three are close to free. The last one is real work, but it is also where an application engineer has the most direct control. On a long-context RAG product, it is often the difference between fitting on the GPU you have and paying for a bigger one.
\nNone of this is exotic. The KV cache is the transformer trading repeated compute for stored state. The memory it costs is one short formula multiplied by two numbers you set: how long the context is, and how many requests you batch. Once you can see those numbers, CUDA out of memory reads as a budget you overspent rather than a mystery. I did this arithmetic often on the HPC and GPU side of the CMS workflow tooling I maintained, and it is the same check that keeps Archi answering long questions without falling off the card.
Diagrams by M. Hassan Ahmed, released under CC0. No external image was used in this post.
","frontmatter":{"title":"LLM KV Cache: Why GPU Memory Runs Out","date":"2026-07-12T00:00:00.000Z","description":"A long prompt or a long agent run can hit CUDA out of memory, and the KV cache is usually why. How it grows, the per-token math, and how to shrink it.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAAB2UlEQVQoz0WO63KbMBBG+dUA5qogkEASMverb6FxYgjGaTLt+z9R12Y6nTmjWe1+ZyWlEV0j+lb0VVSf6/NtuI27aRlu8/G6vCzT/gOuH4cZ+tfTchs+T8VQsQbyICo/bP70wAzyMDtl/Zh27/VpSbtLvpuy/gIFNPPdWB7mbvik6dHAGeRBVAxXrFjPWwPFG4dvXPFkRaoVwak7HEbQgQLQbLZBMcTWvnJPPyoTumvU4Zw3Iu622x4FGawwHhnAQTHzU+ZnAU7AV3Sb6XYEAynaRDQIbzUrIkEW8zaJ4btHGaSWK3wvDrDwPOFhiX0JJ/xCeZjM8qRqR8hmDS3GpPtdHOe0/y52X2XHCSMBIyH3I+6HMoxzJvNQZJoVKppJTZcjPyvLw9d0/TMt03B52Q+n/dC1Q1IcnLhGsrF5bbDaCguHpm6YWVjqJlU0i+p2aCLB8jfRz0E94nwkxUTyO7SYwnIKkjccD748O0GlGgG8BxYALxO461boxYMnXrH4ycpZtr9EfYWC14tobiS7ePLVT94dUt9lMME3CciwCWSKaIXCBnBI5dLGizqP9c9Ra5PSJRWiNYxsnGqGvyqAohr+ytMGr6gbb+NQEzELcdONVAP/H/0Lr/wFUNZLWBRwh3EAAAAASUVORK5CYII=","aspectRatio":1.899441340782123,"src":"/static/3532ef527ffd15a7acd37770357c598c/40a76/hero.png","srcSet":"/static/3532ef527ffd15a7acd37770357c598c/c972b/hero.png 340w,\n/static/3532ef527ffd15a7acd37770357c598c/27625/hero.png 680w,\n/static/3532ef527ffd15a7acd37770357c598c/40a76/hero.png 1360w,\n/static/3532ef527ffd15a7acd37770357c598c/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-07-12-llm-kv-cache-gpu-memory/","previous":"blog/2026-07-09-hnsw-vector-search-explained/","next":"blog/2026-07-11-llm-api-rate-limits-retries-backoff/"}},"staticQueryHashes":["32046230"]}