How PagedAttention Powers vLLM's KV Cache
A self-hosted LLM server wastes most of its GPU memory to KV cache fragmentation. Here is how PagedAttention in vLLM pages the cache like an OS.
A self-hosted LLM server wastes most of its GPU memory to KV cache fragmentation. Here is how PagedAttention in vLLM pages the cache like an OS.