Why KV usage grows

A bigger context window is not a bigger brain. It is a larger working set you must store and re-read on every generated token. KV bytes scale with tokens × layers × KV-heads — so “128k context” is a larger tax, not a discount.
cached past (grows with T) K V K V K V … K V new token Q one decode step reads ALL of it × every layer · × every KV head · × every live request weights — fixed loaded once KV — grows with T re-read every token a bigger window lengthens this bar
Architecture — the ladder of KV reduction
KV resident
—
vs 140 GB weights
—
Bytes read / decode token
—
KB / prompt token
—
Resident memory — weights stay put; KV is the growing term

Weights are loaded once. KV is allocated per live request and re-read in full for every new token. That is why decode gets slower as the conversation grows, even when the model’s advertised window is “plenty.”

Three rungs — only the last cancels T
1 · shrink H GQA · MQA · MLA same tokens, fewer heads 2 · cap T window · sinks · evict drops the middle 3 · don’t store T files · pin · compact harness-side cut

Prefix reuse is none of these: it skips recompute across requests. It does not shorten the bar above.

RungCutsLeaves
GQA / MQA / MLAH (heads or latent)T still linear
Window / sinks / H2OT storedthe dropped middle
NSA / DSAT attended (compute)T stored, unless you also offload
KDA hybridT in the linear layersMLA layers still linear in T
IndexCache / IndexShareindexer FLOPs across layerscore k and stored T
KV quantbytes / elementT still linear
Mooncake / prefix reuseprefill across requeststhis request’s resident KV
Externalize / pinT in the windowlossy unless governance is pinned
What to try
  • Set context to 8k (GQA). Note KV vs 140 GB weights. Drag to 128k — KV crosses the weight bar. The window got “larger”; the GPU got poorer.
  • Click MHA (64 heads) at 32k, then GQA (8). Same tokens, 8× less KV. That is why Llama-class models ship GQA — not because 128k is free.
  • Raise batch to 8 at 32k GQA. Eight conversations, eight caches. Concurrency is a KV problem before it is a FLOP problem.
  • Sliding window 4k at 128k advertised: stored T caps. Bandwidth stops growing. Recall of the dropped middle is a different lab.
  • Then look vs store: DSA shrinks the yellow (softmax) bar, not this teal occupancy bar. KDA is the occupancy bet.
Ask first: what grows when I add a token? A K/V slice, every layer, then the whole cache is streamed on the next decode step. A 1M window licenses a 1M-long tensor. Cut H (MLA/GQA), cap T, or don’t put the token in the window. Sparse (DSA) is not a cut of this bar — it cuts look, not store. That split is look vs store.