![\"The](\"/db510d33194c930b0c99d03b73b96968/static-vs-continuous.svg\")
You put a model behind an API, send it a handful of requests, and watch GPU utilization sit at something embarrassing like 30%. The card is expensive and mostly idle. The first instinct is to batch requests together. That helps, but only up to a point. The reason it stalls is subtle, and fixing it is where most of the throughput on a serving GPU actually comes from.
\nThis post is for engineers running an LLM behind a real endpoint. It explains why a simple batch loop leaves so much performance on the table, what continuous batching does instead, and which settings control it. I ran into this on the GPU side of the CMS workflow tooling I maintained at CERN. I also hit it on Archi, our RAG (retrieval-augmented generation) copilot, where answers vary wildly in length and requests arrive whenever an operator has a question.
\nText generation is autoregressive: the model produces one token, appends it to the sequence, and feeds the whole thing back in to produce the next. So a request is not one GPU call. It is hundreds, one per output token. The very first pass, which processes the entire prompt at once, is called prefill. Every step after that produces a single token and is called decode.
\nNow imagine the obvious batching scheme: collect a few requests, run them as a batch, and return the results when the batch finishes. The catch is that the requests in a batch almost never finish together. One user asks for a yes/no answer and is done in 20 tokens. Another asks for a full explanation that runs for 800.
\nIn this scheme, the short request can’t leave the batch early. Its slot stays pinned until the longest request in the batch is done, and for those hundreds of extra steps it contributes nothing. The GPU is doing work, but a chunk of the batch is just padding.
\n
This is head-of-line blocking: requests stuck behind the slowest one in the batch. It gets worse as the spread in output lengths grows. You can’t batch your way around it by grouping similar requests, because output length is unpredictable. You don’t know how long a request will run until it stops.
\nThe fix came from a 2022 paper, Orca, which changed where the scheduler sits. Instead of scheduling a whole request at a time, it schedules one iteration at a time, where an iteration is a single forward pass of the model. The scheduler runs between every token.
\nThat sounds like a small change, and it is not. Once the scheduler can decide on every step, three things happen the moment a request emits its stop token:
\nNobody waits for the slowest sequence any more. The batch’s membership changes constantly, which is why the technique is usually called continuous batching. (Orca’s own term is iteration-level scheduling; the industry mostly says continuous batching.)
\n![\"One](\"/09563ff330211b36e08a090b7ecff662/scheduler-loop.svg\")
The loop underneath is simple to picture. Each pass evicts finished requests and gives back their memory, admits waiting requests while they fit, and then runs exactly one forward pass:
\nrunning = [] # requests currently in the batch\nwaiting = deque() # requests that have arrived but not started\n\nwhile running or waiting:\n # 1. drop anything that hit its stop token or max length\n finished = [r for r in running if r.is_done()]\n for r in finished:\n r.release_kv_blocks() # give the memory back\n r.return_to_client()\n running = [r for r in running if not r.is_done()]\n\n # 2. pull in new work while there is room on the card\n while waiting and can_admit(waiting[0], running):\n running.append(waiting.popleft())\n\n # 3. run exactly ONE forward pass over the current batch\n step(running) # prefill for new members, one decode token for the restNotice that step 3 mixes two kinds of work in one pass: new members get their prefill while everyone else gets one decode token. Real engines like vLLM and Hugging Face’s Text Generation Inference (TGI) are far more careful than this, but the shape is right: evict, admit, run one step, repeat.
\nAnyscale benchmarked this against the request-at-a-time approach. They reported up to 23x more throughput on some workloads, with lower median latency at the same time, because requests stop queueing behind unrelated long ones.
\nTwo details make the naive picture wrong. Both are worth knowing before you tune anything.
\nBatching a transformer isn’t uniform across the model. Attention for each request depends on that request’s own past tokens, which live in its own KV cache (the stored keys and values from earlier tokens). So you can’t naively stack sequences of different lengths into one matrix for the attention step, the way you can for the big feed-forward layers.
\nOrca’s answer was selective batching: batch the operations that don’t care about sequence shape, and handle attention per sequence. That is what lets requests at completely different positions share a single forward pass at all.
\nThe second detail is memory, and this is the one that bites in production. When you admit a new request, you commit to holding its KV cache for as long as it runs, and that cache grows with every token. Admit too many requests and a batch that fit a moment ago runs out of VRAM mid-generation.
\nSo continuous batching needs a memory manager underneath it. The manager has to pack caches tightly and, when the card fills, temporarily evict a request and bring it back later. In vLLM that manager is PagedAttention, which stores the cache in fixed-size blocks so space can be handed out and reclaimed on demand. Continuous batching is the scheduler; paged memory is what makes the scheduler’s decisions safe.
\nThis is the failure mode that surprises people most. Prefill and decode have very different shapes. A decode step does one token per request and is cheap. A prefill processes the entire prompt at once and, for a long prompt, is a heavy burst of compute.
\nSuppose your scheduler runs a new request’s full prefill in a single iteration. That iteration takes a long time, and every request already streaming tokens to a user goes quiet while it runs. The user watching text appear sees it freeze mid-sentence. On a busy server with long prompts, the stream stutters instead of flowing smoothly.
\n![\"Why](\"/cc098455ed15233b7e9a8c796d0017ca/chunked-prefill.svg\")
The common fix is chunked prefill. It splits a long prompt’s prefill into smaller pieces and interleaves them with everyone else’s decode steps, so ongoing streams keep getting tokens instead of stalling for one giant step. You trade a slightly slower prefill for a much steadier time-between-tokens across all active requests. For an interactive product, that is usually the right trade. vLLM exposes this directly, and the Sarathi-Serve paper explains the design, framing it as an explicit throughput-versus-latency knob.
\nIf you run vLLM, two engine arguments control most of the behavior above:
\nmax_num_seqs caps how many requests can be in the running batch at once. Higher means more throughput and more KV memory pressure.max_num_batched_tokens caps how many tokens (prefill plus decode) one iteration may process. This is the lever that tames prefill bursts. Lower it and long prefills get chunked more aggressively, which smooths latency at some cost to raw prefill speed.There is no single correct setting. An offline batch job wants both cranked up for throughput. An interactive chat endpoint wants max_num_batched_tokens low enough that no single prefill can freeze the streams. Either way, these are the two numbers to reach for, and they trade the same two quantities, throughput and latency, against each other every time.
When a serving GPU is underused, I don’t start by buying a bigger card. I check whether the engine is actually doing continuous batching. A surprising number of homegrown loops still batch a whole request at a time and pin short requests behind long ones. Fixing that is usually the largest win available, the same way paging the KV cache is the largest memory win.
\nAfter that, in order:
\nmax_num_seqs from the length distribution you measured, not one you guessed.None of it is exotic. Continuous batching is one idea (move the scheduler inside the token loop), and almost everything else is bookkeeping to make that idea safe on a finite card.
\nI apply the same reasoning to the tool loops and streaming endpoints behind Archi. The model is rarely the bottleneck; the scheduling around it usually is. Getting a fully loaded GPU is mostly a matter of never letting a slot sit idle while work is waiting for it.
\nDiagrams by M. Hassan Ahmed, released under CC0. No external image was used in this post.
","frontmatter":{"title":"How Continuous Batching Speeds Up LLM Serving","date":"2026-08-31T00:00:00.000Z","description":"A naive LLM server holds finished slots idle until the slowest request in the batch drains. Continuous batching refills them every step. How it works.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAAB80lEQVQozz2QWW+jMBSFeZnOJBAWgzEYzGawzRoITZu0TZqlM9K8jDTz2v//P+Y2lSp9ujqWfHyPj9YkXZv0QB23x+l4ub/AfFkfrg/X8/ZyWB8v2+v5/nKaT6+b02k+77snFTdwH4zancXurPibGRmY581OTce83fFuX29eQfP2SY3HangW6wPManjh/ZOOC7AAmuEkKze3cYkCYXnc9LiFSx1l3y22sNKlcwNlSyf7nDrKV5jrbm6gTNNtZuHKizov7hBtbqIngUrjPmV9wvrAl4EvKJFARBTwKRxcaUsrNtzCcDk8qaMPAQjazMVmKqYh6Qsn5y6P7CxyCmpnofUBQwUlAsyR6ZWWL5ywdmlr+sIN6iHur6H8zcf33dsfOZ+pmmgTp2Mz/+y3v5r5jcZDiEttaVL4pAXZ8k1UPthEWkTyQF0j9U/dv0+Hv2o+hUL60qVdUD4wuafVzqcdBfPCDFduYROF2YDZGvZDitQXPS47XL6y7jFUI5ElrmKvyoliWGDIjHjoV9piFRgoBwMKGygMIkAQ5lXULSMsGJYZqauwHbOppm1BVOZLIPGqj9iLFYHCVl51q6oETCwSIjv+OMrn9Y1e7FW+FWxUyQTU2aaKBgLmH4YPnelO8sXSYSZKkZs6KAEQ6BuOk3wBR92O/wOC1lN8rolhEwAAAABJRU5ErkJggg==","aspectRatio":1.899441340782123,"src":"/static/3113e5e31d17506d62c29098723d1894/40a76/hero.png","srcSet":"/static/3113e5e31d17506d62c29098723d1894/c972b/hero.png 340w,\n/static/3113e5e31d17506d62c29098723d1894/27625/hero.png 680w,\n/static/3113e5e31d17506d62c29098723d1894/40a76/hero.png 1360w,\n/static/3113e5e31d17506d62c29098723d1894/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-08-31-continuous-batching-llm-serving/","previous":"blog/2026-09-01-metadata-filtering-rag-vector-search/","next":"blog/2026-09-04-idempotency-keys-fastapi-safe-retries/"}},"staticQueryHashes":["32046230"]}