Inference Engineering: What an Inference Engineer Does
Inference engineering is serving trained models fast, cheaply and reliably. What an inference engineer does: the metrics, the levers and the skills.
#LLM(72)
Splitting an LLM Across GPUs: Tensor vs Pipeline
A model too big for one GPU has to be split. How tensor and pipeline parallelism divide an LLM, what each costs, and when to reach for which.
RoPE: How LLMs Encode Position and Extend Context
How rotary position embeddings encode token order by rotating query and key vectors, and how position interpolation and YaRN stretch a model's context window.
How Much GPU Memory to Serve an LLM
Pick a GPU for an LLM and it 'fits' on paper, then OOMs under load. A quick way to estimate the VRAM a model needs: weights, KV cache, and overhead.
LLM Model Routing: Send Easy Prompts to Small Models
Most prompts hitting an LLM app are simple. Route those to a small model and escalate only the hard ones to a large model to cut cost and keep quality.
Mixture of Experts: Sparse Compute, Dense VRAM
Mixture of Experts routes each token to a couple of expert layers, so the model runs cheaper per token yet still needs every expert sitting in GPU memory.
Dedupe a RAG Corpus with MinHash and LSH
Exact md5 hashing misses near-duplicate documents; comparing every pair is O(n²). How MinHash and LSH find and cluster them across a RAG corpus.
Fix Unnecessary React Re-Renders in a Chat UI
An LLM streams tokens and React repaints your whole chat history on every one. How to keep the message list still while a single reply streams in.
RAG Citations: Make the LLM Cite Its Sources
How to make a RAG system cite its sources: force the model to reference chunks by ID, then verify each citation against the source text before trusting it.
Agentic RAG: When Should It Retrieve?
Naive RAG retrieves on every turn, even when it shouldn't. How agentic RAG lets the model route, grade its own context, and re-retrieve, plus the tradeoffs.
FastAPI WebSockets for Streaming LLM Chat
SSE streams one direction only. When an LLM chat needs the browser to interrupt a running response, FastAPI WebSockets give you a full-duplex channel.
Maximal Marginal Relevance for RAG
Top-k vector search keeps returning near-duplicate chunks that fill the context window. How Maximal Marginal Relevance reranks for relevant, diverse results.
Parent Document Retrieval: Search Small, Answer Big
Small chunks search precisely but read like fragments. Parent document retrieval matches on child chunks, then feeds the whole parent to the model.
Parsing PDFs for RAG: Text, Tables, and OCR
How to parse PDFs for a RAG pipeline: detect born-digital vs scanned files, pull out tables without flattening them, and reach for OCR only when you must.
Parsing Partial JSON While an LLM Streams
LLM structured output isn't valid JSON until the last token. Here's how to complete and parse a streaming buffer so you can render fields as they arrive.
How Continuous Batching Speeds Up LLM Serving
A naive LLM server holds finished slots idle until the slowest request in the batch drains. Continuous batching refills them every step. How it works.
Prompt Caching with the Claude API
Prompt caching reuses a stable prompt prefix to cut Claude API cost and latency. How breakpoints, TTLs, and cache reads work, and what quietly breaks a hit.
Circuit Breakers for LLM API Calls
When an LLM provider degrades, retries make it worse. A practical guide to adding a circuit breaker in Python: the three states, tuning, and failure modes.
Contextual Retrieval for RAG, Step by Step
Vector search misses chunks that lost their document context. Contextual Retrieval prepends a short LLM-written summary to each chunk before you embed it.
Autoscaling a FastAPI Service with the Kubernetes HPA
The Kubernetes HPA scales pods from a metric, but its defaults thrash and lag under real load. How the control loop works, how to tune it, and where it breaks.
Scheduling GPU Pods on Kubernetes
A GPU node won't run your Pods until a device plugin advertises it. How Kubernetes discovers GPUs, how to share one across Pods, and where scheduling breaks.
Speculative Decoding: Faster LLM Inference
Speculative decoding uses a small draft model to guess tokens a big model verifies in one parallel pass, cutting LLM latency with no change to output.
Prefill vs Decode: Two Phases of LLM Inference
The first token is slow, the rest stream fast. Why LLM inference splits into a compute-bound prefill and a memory-bound decode, and what it costs you.
Tracing a RAG Pipeline with OpenTelemetry
Trace a RAG pipeline with OpenTelemetry: instrument FastAPI, put each retrieval and model step in its own span, and read the latency waterfall in OpenSearch.
HyDE: Hypothetical Document Embeddings for RAG
Vector search fails when a short question looks nothing like its answer. HyDE has an LLM draft a fake answer, embeds that, and retrieves against it instead.
Prompt Injection Defense for RAG Systems
Retrieved documents are untrusted input. A practical guide to defending a RAG copilot against direct and indirect prompt injection, and why filters alone fail.
LLM-as-a-Judge: Scoring RAG Answer Quality
Retrieval metrics say the right docs came back, not that the answer is right. Build an LLM-as-a-judge to score RAG answers for faithfulness and quality.
Fan Out Concurrent LLM Calls with asyncio.gather
Awaiting retrieval and LLM calls one by one wastes seconds per request. Here's how to fan them out with asyncio.gather, bound it, and handle partial failures.
Build a Semantic Cache for LLM Apps
Semantic caching for LLM apps: cache answers by embedding similarity in FastAPI, tune the cutoff, and avoid false cache hits. Working code and failure modes.
How PagedAttention Powers vLLM's KV Cache
A self-hosted LLM server wastes most of its GPU memory to KV cache fragmentation. Here is how PagedAttention in vLLM pages the cache like an OS.
How Grouped-Query Attention Shrinks the KV Cache
Grouped-query attention shares key/value heads across query heads to cut the KV cache. How GQA sits between MHA and MQA, with the memory math and code.
Rate Limiting a FastAPI Service with a Token Bucket
Add per-user rate limiting to a FastAPI backend with the token bucket algorithm: an in-process version, an atomic Redis script, 429s, and the failure modes.
Matryoshka Embeddings: Smaller Vectors for RAG
Matryoshka embeddings front-load meaning into a vector's first dimensions, so you can truncate them for cheaper, faster RAG search without losing much recall.
How FlashAttention Speeds Up the Attention Layer
Attention is memory-bound, not compute-bound. Here's how FlashAttention uses tiling and online softmax to skip the N×N matrix and run exact attention faster.
How Byte-Pair Encoding Tokenizes Text for LLMs
Byte-pair encoding turns text into the tokens an LLM bills and reasons over. How BPE merges are learned, why token counts drive cost, and where it breaks.
Late Interaction Retrieval for RAG with ColBERT
A bi-encoder averages token detail away; a cross-encoder is too slow to rank a corpus. Late interaction with ColBERT sits between them. Here is how it works.
Contextual Retrieval for RAG Pipelines
Chunking a document for RAG strips each piece of its context. Contextual retrieval adds an LLM-written note to every chunk before you index it.
Metadata Filtering in Vector Search for RAG
Adding a metadata filter to a vector search can silently return fewer results or wreck recall. How post-filter, pre-filter, and filterable HNSW actually differ.
Graceful Shutdown for FastAPI on Kubernetes
A rolling deploy sends SIGTERM and kills your FastAPI pod mid-request, dropping live SSE streams. How to catch it, drain connections, and shut down cleanly.
Managing the Context Window in Long Agent Runs
An LLM agent that runs long enough fills its context window and starts to slow or fail. How to prune, compact, and offload context so agents keep going.
LLM Sampling: Temperature, Top-p, and Top-k
Temperature, top-p, and top-k are the three knobs that shape how an LLM picks each token. How each one works, when to reach for it, and how they interact.
How to Choose an Embedding Model for RAG
The embedding model sets the ceiling on RAG retrieval quality. How to choose one by task fit, sequence length, dimensions, and domain, plus the silent bugs.
Prompt Caching: Cut LLM Cost and Latency
Prompt caching reuses a request's prefix to cut LLM cost and latency. How the prefix match works, where to put the breakpoint, and the silent cache misses.
LLM Quantization: INT8, GPTQ, and AWQ Explained
The weights don't fit on the GPU you have. How LLM quantization shrinks them to INT8 or INT4, why GPTQ and AWQ beat naive rounding, and where it breaks.
How Speculative Decoding Speeds Up LLM Inference
Speculative decoding uses a small draft model to guess tokens a big model verifies in one pass, cutting LLM latency 2-3x without changing the output.
Query Rewriting for Better RAG Retrieval
Short, vague, follow-up questions don't match how your docs are written. How query rewriting, multi-query expansion, and HyDE fix retrieval before it runs.
Tracing an LLM Agent with OpenTelemetry
An LLM agent request hides where the time and tokens went behind one flat log. Trace it with OpenTelemetry spans, the GenAI conventions, and where it breaks.
Continuous Batching for LLM Inference
Static batching leaves the GPU idle when requests finish at different steps. How continuous batching schedules LLM inference per token to raise throughput.
LLM-as-a-Judge: Evaluating LLM Output Quality
Human review does not scale for grading LLM answers. How to use an LLM as a judge: write a rubric, score with structured output, and control the biases.
Indirect Prompt Injection in RAG Systems
A RAG copilot reads tickets and logs, so whoever writes them can plant instructions in the prompt. How indirect prompt injection works and how to contain it.
How to Build a Semantic Cache for LLM Apps
Exact-match caching misses paraphrases, so LLM bills stay high. Here is how to build a semantic cache with embeddings, a similarity threshold, and its traps.
Build an MCP Server for Your LLM Agent
MCP standardizes how LLM agents reach your tools and data. A hands-on guide to building an MCP server in Python, picking a transport, and where it breaks.
Cancelling an LLM Stream in React with AbortController
A user hits stop or switches chats and the old LLM stream keeps writing tokens and running up cost. How to cancel a streaming fetch in React the right way.
Getting Reliable JSON Out of an LLM
Getting an LLM to return JSON is easy; getting valid JSON every time is not. How to use JSON Schema, constrained decoding, and validation to make it reliable.
Streaming LLM Markdown in React Without Flicker
LLM tokens arrive one at a time, and re-parsing Markdown on every token flickers and drags. How to render streaming Markdown in React without the jank.
LLM KV Cache: Why GPU Memory Runs Out
A long prompt or a long agent run can hit CUDA out of memory, and the KV cache is usually why. How it grows, the per-token math, and how to shrink it.
Handling LLM API Rate Limits: Retries and Backoff
Your LLM backend returns 429s the moment traffic bursts. How to retry with backoff and jitter, respect Retry-After, and pace fan-out to stay under the limit.
FastAPI Event Loop Blocking: Sync vs Async
One blocking call in a FastAPI route stalls every other request, including live SSE streams. Here is how the event loop breaks, and how to keep it free.
How HNSW Vector Search Actually Works
Every RAG stack leans on HNSW but treats it as a black box. Here is how the layered graph index finds nearest neighbors fast, and the knobs that matter.
Cross-Encoder Reranking for RAG Pipelines
Retrieval puts the right chunk at rank 8, but the generator only reads the top few. How a cross-encoder reranker reorders RAG candidates, and where it fails.
Chunking Strategies for RAG Pipelines
How to chunk documents for a RAG pipeline: why fixed-size splitting fails, structure-aware splitting, size and overlap tradeoffs, and the failure modes.
Running LLM-Generated Code in a Sandbox
An LLM that writes and runs code needs real isolation, not a try/except. How to sandbox AI-generated code with E2B microVMs, and the failure modes.
Measuring RAG Retrieval Quality: recall@k, MRR
Changed your embeddings or added a reranker? Measure it. Build a golden set and score retrieval with recall@k, MRR, and nDCG before you trust the change.
Hybrid Search for RAG: BM25 + Vectors
Vector search alone misses exact IDs and error codes. Here's how to combine BM25 keyword search with dense retrieval, fuse the rankings with RRF, and rerank.
Streaming LLM Responses from FastAPI with SSE
Stream LLM output token by token from a FastAPI backend with Server-Sent Events: working code, the EventSource client, proxy buffering, and failure modes.
How an LLM Agent Tool-Calling Loop Works
A practical look at the agent loop behind LLM tools: how the model asks to call a tool, your code runs it, and the result feeds back until the answer is done.
Incremental Indexing for RAG Pipelines
How to keep a RAG index fresh without full rebuilds: detect changed files with checksums, re-embed only what changed, and handle deletions safely.
CloudCanvasAI
AI-powered document creation platform pairing Claude with a live .docx / .pptx / .pdf / .xlsx preview. Chat on the left, documents render live on the right, each user gets their own isolated E2B sandbox.
Argusa AI Challenge 2025 · Runner-Up 🥈
Secured 2nd place at the Argusa AI Challenge 2025 by building ARGRAG, a RAG system for complex enterprise document corpora with multi-modal analysis, metadata intelligence, and real-time performance.
Archi · LLM Copilot for CMS Ops
Partnering with MIT to build an LLM-powered RAG copilot for CMS computing operations.
LLM DevMate
A VSCode extension that streamlines pasting code context into LLMs. One-click copy, token-aware, works with OpenAI and other providers.
Gemini Alchemy
A Gemini-powered tool that turns abstract ideas and complex concepts into visually appealing mindmaps. FastAPI backend, Streamlit frontend.