Attention, Controlled — MLA vs MHA, MQA, GQA on a from-scratch GPT

About this series

For long-context inference the KV cache, not the parameter count, is the wall: it grows linearly with context and dominates both memory and memory-bandwidth at decode time. Every modern attention variant—Multi-Query (MQA), Grouped-Query (GQA), and DeepSeek-V2's Multi-head Latent Attention (MLA)—is, at bottom, a different bet about how to shrink that cache without giving up too much.

Public comparisons usually confound the attention mechanism with changes in depth, width, data, or training recipe. This series removes that confound. Every variant shares one fixed backbone—identical layers, width, RoPE, SwiGLU MLP, optimizer, schedule, data, and token budget—and only the attention module changes. Whatever differences appear are therefore attributable to the attention mechanism alone. Each paper then pushes that controlled comparison along a different axis: first efficiency and quality, then capability.

The papers

1 The Attention Variant Is the Only Variable

At a fixed compute and parameter budget, how much quality does each KV-cache-reduction strategy actually cost, and what does it buy back in inference efficiency?

At 124M parameters, cutting the cache is nearly free for language-model quality: the four variants land within ~4.6% perplexity. MLA compresses the KV cache 5.6× vs MHA and Pareto-dominates GQA — lower perplexity and a smaller cache — trailing full MHA by only ~1.8% perplexity while—via weight absorption—overtaking it on decode throughput at long context.

Read →

2 The Recall Cliff: Where KV-Cache Reduction Finally Costs Capability

Perplexity says the four caches are nearly equal — but does the smaller cache cost capability? Under associative-recall load, where does each variant break?

On dense associative recall there is a capacity cliff ordered by cache width (MHA > GQA > MQA > MLA): past ~64 stored pairs accuracy collapses, and at 96 pairs only caches ≥1024 B escape while ≤512 B stay capacity-bound. The cliff is invisible to natural-text perplexity—and two confound controls show its steepness is partly a learning-rate-schedule artifact, though its floor is genuinely cache-set.

Read →