![\"The](\"/f6c66733bd513b5e176710fe3ac4354d/temperature-softmax.svg\")
The first time my RAG copilot returned a subtly wrong factual answer, I blamed retrieval. But the right document was in context. The model had the source in front of it and still drifted into a plausible-sounding detail that the source did not contain. Retrieval was fine. The problem was the generation config: temperature=0.8, copied from an example tuned for chat. That one number did more damage than any missing chunk.
Sampling parameters are the part of the stack most people set once and never revisit. They sit in a config file next to the model name, get copied between projects, and quietly decide how adventurous every token is.
\nThis post is for engineers who call an LLM as one component of a system and want to know what temperature, top_p, and top_k actually do to its output, instead of treating them as three dials to jiggle when something looks off. I start with how a model picks a token, show where each knob acts, and then spend time on how the knobs interact and the failure modes that bite in production.
A language model does not emit words. At each step it produces a vector of logits: one raw score per token in its vocabulary, which means tens of thousands of scores. These scores are unbounded, so they are not probabilities. To get something you can sample from, you pass them through softmax. Softmax exponentiates each score and then normalizes, so the results sum to one:
\np_i = exp(z_i) / Σ_j exp(z_j)Here z_i is the logit for token i and p_i is its probability. The result is a probability distribution over the whole vocabulary, and every decoding strategy is a rule for picking one token from it:
Everything in the rest of this post either reshapes that distribution or restricts which part of it you may draw from, before the draw happens.
\nTemperature is a single number, T, that divides the logits before softmax runs:
p_i = exp(z_i / T) / Σ_j exp(z_j / T)That division is the entire mechanism. With T below 1, the logits get larger, which widens the gaps between them, and softmax piles even more probability onto the tokens that were already likely. With T above 1, the gaps shrink and probability spreads out toward the tail (the long list of unlikely tokens). The diagram below shows the same six logits at three temperatures.
Two points stand out in that picture.
\nFirst, temperature adds no information. It cannot make the model consider a token it wasn’t already considering. It only moves probability among the candidates the logits already ranked. High-temperature “creativity” is really the model being allowed to pick further down its own list.
\nSecond, the endpoints are special. As T approaches 0, softmax collapses onto the single top token, which is exactly greedy decoding. Most APIs special-case temperature=0 to mean greedy instead of dividing by zero.
The mental model that serves me well: temperature is a confidence knob, not a quality knob. Low temperature makes the model commit to what it already thinks is most likely. You want that for extraction, classification, and the grounded factual answers in a retrieval system. It’s the same instinct behind getting reliable structured output out of a model. High temperature buys variety, but the price is the model wandering into its own long tail, which is where confident nonsense lives.
\nTemperature reshapes the whole distribution but never rules a token out. Even at T=1, a token with 3% probability at the bottom of the list is still a legal draw. Over thousands of tokens, those occasional draws from the far tail are where text goes off the rails. Truncation sampling fixes this by discarding the tail entirely and sampling only from what remains. The two common rules differ in where they cut.
Top-k is the blunt version. Keep the k highest-probability tokens, discard the rest, renormalize the survivors so they sum to one, and sample. top_k=40 means “only ever consider the 40 most likely next tokens.”
It is simple and cheap. Its weakness is that k is a fixed count that ignores the shape of the distribution:
k of 40 drags in 39 tokens the model barely wanted.k may cut off good candidates.Top-p, also called nucleus sampling (Holtzman et al., 2019), adapts to the shape instead. Sort the tokens by probability and walk down the list, adding up probability as you go. Keep the smallest set whose cumulative probability reaches p, then renormalize and sample.
top_p=0.9 means “keep however many tokens it takes to cover 90% of the probability, and no more.” On a peaked distribution that might be two tokens; on a flat one it might be fifty. The cutoff moves with the model’s confidence, which is exactly the property top-k lacks.
![\"Two](\"/0f78cf97c86004949ddec464eef0a79b/topk-vs-topp.svg\")
The paper that introduced nucleus sampling is worth reading for the full motivation. It showed that pure greedy decoding and pure temperature sampling both degrade into repetitive or incoherent text, and that truncating the unreliable tail is what keeps generation both varied and sane. That is why nearly every production stack defaults to top-p.
\nYou rarely set just one knob. In most implementations all three are active at once, and the order in which they apply matters. The common pipeline, which is also the default in Hugging Face’s transformers, runs like this:
Here is that sampling step as pseudocode, without the tensor plumbing. The numbered comments match the order above.
\n# The shape of what a sampling step does, minus the tensor plumbing.\nlogits = model_step(context) # one score per vocab token\nlogits = logits / temperature # 1. reshape\nlogits = keep_top_k(logits, k=40) # 2. hard cap on count\nlogits = keep_top_p(logits, p=0.9) # 3. adaptive cap on mass\nprobs = softmax(logits) # 4. renormalize survivors\ntoken = sample(probs) # 5. drawReading it as a pipeline explains two things.
\nFirst, combinations that look redundant are not. top_k=40 with top_p=0.9 is not two ways of saying the same thing. Top-k sets a hard ceiling on how many tokens can ever be considered, and top-p tightens that further whenever the model is confident.
Second, it exposes a trap. If you set temperature=0 for deterministic output, top-p and top-k are irrelevant, because greedy decoding already ignores everything except the top token. Leaving them in the config is harmless, but it misleads the next person who reads it.
Temperature 0 does not guarantee identical output. Greedy decoding is deterministic in theory, yet the same prompt sent twice to a hosted model can return different text. The logits aren’t bit-identical from run to run because of floating-point non-associativity (the order of additions changes the rounding), changing batch sizes, and routing differences in mixture-of-experts models. When the top two tokens are close, a tiny perturbation flips which one wins. If your tests assert exact string equality on model output, they will flake. Assert on the parsed result instead.
\nLow temperature plus aggressive truncation produces loops. Set temperature low and set a tight top_k or top_p, and you can starve the distribution down to one or two tokens at every step. The model gets stuck repeating a phrase because you removed every alternative it might have escaped through. If output degenerates into repetition, the fix is usually to loosen truncation, not tighten it.
Inherited defaults were tuned for a different job. An SDK example might ship temperature=0.7. That is reasonable for open-ended chat and wrong for a classifier whose label your code parses. When the model is one component feeding another, “close enough” is a bug, and the temperature that felt fine in a demo is why your extractor occasionally invents a field. That was exactly my RAG bug: a chat-tuned temperature on a task that needed the model to stay pinned to its source.
Some frontier models no longer expose the knobs. This matters when you pick a provider. The newest Anthropic models drop temperature, top_p, and top_k entirely and reject requests that set them, steering you toward prompting and other controls instead (see the Claude Opus 4.7 migration notes). If your pipeline depends on a specific temperature, that is a real portability cost. Don’t assume every model exposes the same dials.
When code consumes the output, I default to temperature=0 and stop there. That covers extraction, classification, tool-argument generation, and the grounded answer step in a RAG pipeline. Determinism is worth more than variety here, and the truncation parameters don’t matter under greedy decoding anyway. This is the setting I used for the factual paths in Archi, the retrieval copilot I worked on for CMS computing operations at CERN. In ops answers, a fabricated detail is worse than a blunt one.
When variety is the point (drafting, brainstorming, generating diverse candidates to rank later), I start from a moderate temperature around 0.7 with top_p=0.9 and adjust from there. I treat temperature as the coarse knob and top-p as the safety rail that keeps the tail out. Newer alternatives such as min-p sampling are worth watching for high-temperature settings, but top-p remains the dependable default.
Mostly, though, the advice is smaller than any specific number: read the sampling config as carefully as you read the prompt. It is part of the model’s behavior, not boilerplate. The difference between a grounded answer and a confident hallucination is sometimes a single division by T. If you’re wiring a model into a larger system, the same care applies one layer up, in the agent tool loop, where a stray token can send a whole chain of tool calls the wrong way.
Diagrams by M. Hassan Ahmed, released under CC0. No external image was used for this post; the figures are original work by the author.
","frontmatter":{"title":"LLM Sampling: Temperature, Top-p, and Top-k","date":"2026-07-29T00:00:00.000Z","description":"Temperature, top-p, and top-k are the three knobs that shape how an LLM picks each token. How each one works, when to reach for it, and how they interact.","thumbnail":{"childImageSharp":{"fluid":{"base64":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAIAAADwazoUAAAACXBIWXMAAAsSAAALEgHS3X78AAABzklEQVQoz21QyY6bQBTkaKChVxbTzTbYbrMZMs4otsGeUXKKMusll/xA/v8L8tqWIkWKVGq9qq63WlRPAF7P2/P79vKhz+/V8Xl1egG6Oj3r89tmfl1PLyDWjx/r6XUzvwEVzZnqk+X4ietLx1tWej/un7phboZp2x0B7TD346XZTe1u0u2hNfoB9Gpzj7CERMtl2RW5jdUCpzbJXJo7Bpl5r9QmKdDra/SFryCAFMsFCSsSlLLoSihJUxcrgIMlYlmU1h41BrCBiILKW9Y47b2kcWlmOcan/GDNZM/lzmOFy0sUrqE2i3SQdCSuEcmcWzleEtnRbPSDFVCzsycqKkeqRiIHmvSuGTgN0kFUR7Y6iuIzi2sbToMlK/Z8deTVAQLECgtO5YeaLLv49FO0X0nceLx0cELjhuuL+v6bryeS9NAD0ZyqXfjpx/LxFy0ecKSthRcjfkflwIoHlt0bH1GOH4ukY9mebebg7gt0BhvMybJR1E9B+43n0Dm3bC8GwA7QE8c1zIyp4gKWTCBHyB6H2sXJ1RbBOUnSmp1FZaMIkiMAFjkN7wAeVTcFUcXCkkUVDcu/IgQkKACY50CtBQoBtz/AAvCv8j/RDAv0D2etR5Bw7PryAAAAAElFTkSuQmCC","aspectRatio":1.899441340782123,"src":"/static/c7670afd8e01157c75428c788f6b297f/40a76/hero.png","srcSet":"/static/c7670afd8e01157c75428c788f6b297f/c972b/hero.png 340w,\n/static/c7670afd8e01157c75428c788f6b297f/27625/hero.png 680w,\n/static/c7670afd8e01157c75428c788f6b297f/40a76/hero.png 1360w,\n/static/c7670afd8e01157c75428c788f6b297f/ed396/hero.png 2000w","sizes":"(max-width: 1360px) 100vw, 1360px"}}}}}},"pageContext":{"slug":"/2026-07-29-llm-sampling-temperature-top-p-top-k/","previous":"blog/2026-07-24-speculative-decoding-llm-inference/","next":"blog/2026-07-28-choosing-embedding-model-rag/"}},"staticQueryHashes":["32046230"]}