Inference Engineering: What an Inference Engineer Does
Inference engineering is serving trained models fast, cheaply and reliably. What an inference engineer does: the metrics, the levers and the skills.
#Inference(15)
Splitting an LLM Across GPUs: Tensor vs Pipeline
A model too big for one GPU has to be split. How tensor and pipeline parallelism divide an LLM, what each costs, and when to reach for which.
RoPE: How LLMs Encode Position and Extend Context
How rotary position embeddings encode token order by rotating query and key vectors, and how position interpolation and YaRN stretch a model's context window.
How Much GPU Memory to Serve an LLM
Pick a GPU for an LLM and it 'fits' on paper, then OOMs under load. A quick way to estimate the VRAM a model needs: weights, KV cache, and overhead.
Mixture of Experts: Sparse Compute, Dense VRAM
Mixture of Experts routes each token to a couple of expert layers, so the model runs cheaper per token yet still needs every expert sitting in GPU memory.
How Continuous Batching Speeds Up LLM Serving
A naive LLM server holds finished slots idle until the slowest request in the batch drains. Continuous batching refills them every step. How it works.
Speculative Decoding: Faster LLM Inference
Speculative decoding uses a small draft model to guess tokens a big model verifies in one parallel pass, cutting LLM latency with no change to output.
Prefill vs Decode: Two Phases of LLM Inference
The first token is slow, the rest stream fast. Why LLM inference splits into a compute-bound prefill and a memory-bound decode, and what it costs you.
How PagedAttention Powers vLLM's KV Cache
A self-hosted LLM server wastes most of its GPU memory to KV cache fragmentation. Here is how PagedAttention in vLLM pages the cache like an OS.
How Grouped-Query Attention Shrinks the KV Cache
Grouped-query attention shares key/value heads across query heads to cut the KV cache. How GQA sits between MHA and MQA, with the memory math and code.
How FlashAttention Speeds Up the Attention Layer
Attention is memory-bound, not compute-bound. Here's how FlashAttention uses tiling and online softmax to skip the N×N matrix and run exact attention faster.
LLM Quantization: INT8, GPTQ, and AWQ Explained
The weights don't fit on the GPU you have. How LLM quantization shrinks them to INT8 or INT4, why GPTQ and AWQ beat naive rounding, and where it breaks.
How Speculative Decoding Speeds Up LLM Inference
Speculative decoding uses a small draft model to guess tokens a big model verifies in one pass, cutting LLM latency 2-3x without changing the output.
Continuous Batching for LLM Inference
Static batching leaves the GPU idle when requests finish at different steps. How continuous batching schedules LLM inference per token to raise throughput.
LLM KV Cache: Why GPU Memory Runs Out
A long prompt or a long agent run can hit CUDA out of memory, and the KV cache is usually why. How it grows, the per-token math, and how to shrink it.