RoPE: How LLMs Encode Position and Extend Context
How rotary position embeddings encode token order by rotating query and key vectors, and how position interpolation and YaRN stretch a model's context window.
How rotary position embeddings encode token order by rotating query and key vectors, and how position interpolation and YaRN stretch a model's context window.
Grouped-query attention shares key/value heads across query heads to cut the KV cache. How GQA sits between MHA and MQA, with the memory math and code.