Speculative Decoding: Faster LLM Inference
Speculative decoding uses a small draft model to guess tokens a big model verifies in one parallel pass, cutting LLM latency with no change to output.
Speculative decoding uses a small draft model to guess tokens a big model verifies in one parallel pass, cutting LLM latency with no change to output.
Speculative decoding uses a small draft model to guess tokens a big model verifies in one pass, cutting LLM latency 2-3x without changing the output.