Prompt Caching with the Claude API
Prompt caching reuses a stable prompt prefix to cut Claude API cost and latency. How breakpoints, TTLs, and cache reads work, and what quietly breaks a hit.
#FastAPI(26)
Read more →
Circuit Breakers for LLM API Calls
When an LLM provider degrades, retries make it worse. A practical guide to adding a circuit breaker in Python: the three states, tuning, and failure modes.
Autoscaling a FastAPI Service with the Kubernetes HPA
The Kubernetes HPA scales pods from a metric, but its defaults thrash and lag under real load. How the control loop works, how to tune it, and where it breaks.
Tracing a RAG Pipeline with OpenTelemetry
Trace a RAG pipeline with OpenTelemetry: instrument FastAPI, put each retrieval and model step in its own span, and read the latency waterfall in OpenSearch.
Fan Out Concurrent LLM Calls with asyncio.gather
Awaiting retrieval and LLM calls one by one wastes seconds per request. Here's how to fan them out with asyncio.gather, bound it, and handle partial failures.
Build a Semantic Cache for LLM Apps
Semantic caching for LLM apps: cache answers by embedding similarity in FastAPI, tune the cutoff, and avoid false cache hits. Working code and failure modes.
Rate Limiting a FastAPI Service with a Token Bucket
Add per-user rate limiting to a FastAPI backend with the token bucket algorithm: an in-process version, an atomic Redis script, 429s, and the failure modes.
Async Connection Pooling in FastAPI
Your async FastAPI app stalls under load, then times out. Here is how SQLAlchemy pool_size and max_overflow really work, and how to size them for Postgres.
How FastAPI Dependency Injection Actually Works
A practical guide to FastAPI dependency injection: how Depends resolves a graph, yield setup and teardown, per-request caching, and where it leaks.
FastAPI BackgroundTasks vs a Task Queue
FastAPI BackgroundTasks run inside your web process and disappear on restart. When that is fine, when you need a real task queue, and how to move over.
RAG Vector Search in Postgres with pgvector
Store and query RAG embeddings inside Postgres with pgvector: HNSW indexing, distance operators, metadata filtering, hybrid search, and the tradeoffs I hit.
Graceful Shutdown for FastAPI on Kubernetes
A rolling deploy sends SIGTERM and kills your FastAPI pod mid-request, dropping live SSE streams. How to catch it, drain connections, and shut down cleanly.
Prompt Caching: Cut LLM Cost and Latency
Prompt caching reuses a request's prefix to cut LLM cost and latency. How the prefix match works, where to put the breakpoint, and the silent cache misses.
Query Rewriting for Better RAG Retrieval
Short, vague, follow-up questions don't match how your docs are written. How query rewriting, multi-query expansion, and HyDE fix retrieval before it runs.
Getting Reliable JSON Out of an LLM
Getting an LLM to return JSON is easy; getting valid JSON every time is not. How to use JSON Schema, constrained decoding, and validation to make it reliable.
Handling LLM API Rate Limits: Retries and Backoff
Your LLM backend returns 429s the moment traffic bursts. How to retry with backoff and jitter, respect Retry-After, and pace fan-out to stay under the limit.
FastAPI Event Loop Blocking: Sync vs Async
One blocking call in a FastAPI route stalls every other request, including live SSE streams. Here is how the event loop breaks, and how to keep it free.
Cross-Encoder Reranking for RAG Pipelines
Retrieval puts the right chunk at rank 8, but the generator only reads the top few. How a cross-encoder reranker reorders RAG candidates, and where it fails.
Chunking Strategies for RAG Pipelines
How to chunk documents for a RAG pipeline: why fixed-size splitting fails, structure-aware splitting, size and overlap tradeoffs, and the failure modes.
Streaming LLM Responses from FastAPI with SSE
Stream LLM output token by token from a FastAPI backend with Server-Sent Events: working code, the EventSource client, proxy buffering, and failure modes.
Incremental Indexing for RAG Pipelines
How to keep a RAG index fresh without full rebuilds: detect changed files with checksums, re-embed only what changed, and handle deletions safely.
CloudCanvasAI
AI-powered document creation platform pairing Claude with a live .docx / .pptx / .pdf / .xlsx preview. Chat on the left, documents render live on the right, each user gets their own isolated E2B sandbox.
Argusa AI Challenge 2025 · Runner-Up 🥈
Secured 2nd place at the Argusa AI Challenge 2025 by building ARGRAG, a RAG system for complex enterprise document corpora with multi-modal analysis, metadata intelligence, and real-time performance.
CMS Workflow Management DevOps
Developing, Operating and extending on WMCore + Unified, so CMS Monte Carlo and reconstruction workloads stay healthy across the WLCG sites.
Archi · LLM Copilot for CMS Ops
Partnering with MIT to build an LLM-powered RAG copilot for CMS computing operations.
Gemini Alchemy
A Gemini-powered tool that turns abstract ideas and complex concepts into visually appealing mindmaps. FastAPI backend, Streamlit frontend.