Inference Engineering: What an Inference Engineer Does
Inference engineering is serving trained models fast, cheaply and reliably. What an inference engineer does: the metrics, the levers and the skills.
#Performance(4)
Fix Unnecessary React Re-Renders in a Chat UI
An LLM streams tokens and React repaints your whole chat history on every one. How to keep the message list still while a single reply streams in.
Speculative Decoding: Faster LLM Inference
Speculative decoding uses a small draft model to guess tokens a big model verifies in one parallel pass, cutting LLM latency with no change to output.
How Speculative Decoding Speeds Up LLM Inference
Speculative decoding uses a small draft model to guess tokens a big model verifies in one pass, cutting LLM latency 2-3x without changing the output.