RoPE: How LLMs Encode Position and Extend Context
How rotary position embeddings encode token order by rotating query and key vectors, and how position interpolation and YaRN stretch a model's context window.
#Transformers(2)
How Grouped-Query Attention Shrinks the KV Cache
Grouped-query attention shares key/value heads across query heads to cut the KV cache. How GQA sits between MHA and MQA, with the memory math and code.