# Learning note — why KV usage grows

Companion to [`sub-apps/sim-kv-growth.html`](../sub-apps/sim-kv-growth.html). Prefix reuse
([the other KV note](kv-cache-prefix-reuse.md)) is a *cross-request* trick. This note is the
*in-request* tax: the tensor you carry, and re-read, because attention is defined over the
past. Every arXiv ID is registered in `harness-research/ledger/ledger.json`.

## The question that generates everything

**What grows when I add one token to the prompt?**

Not “the model gets smarter.” One key vector and one value vector, **per layer, per KV
head, per live request** — then that whole cache is streamed from GPU memory on the
**next** decode step. The formula is linear in every factor; there is no square root, no
amortization from a “larger window.”

```
KV bytes = 2 × L_layers × H_kv × d_head × T_stored × bytes × batch
```

A Llama-3-70B-class GQA stack (80 layers, 8 KV heads, d=128, fp16) adds ~320 KB per
prompt token. At 8k that is ~2.5 GB; at 128k ~40 GB; at 1M ~312 GB — **past the 140 GB
weight footprint**. Without GQA (64 KV heads) multiply by 8. Batch 8 conversations and
you have eight caches.

That is why KV is “a problem”: at long context, **decode is memory-bandwidth bound**.
FLOPs per new token are cheap once K/V exist; moving the growing cache across the bus
is not. A 128k window does not make this cheaper. It licenses a 128k-long tensor.

## Why people confuse window size with “more for free”

Marketing reports maximum *capacity* (how many tokens the position encodings and
kernels will accept). Serving pays for *occupancy* (how many of those slots are filled
on this request). The two moved in opposite directions in 2024–26: advertised windows
went 8k → 128k → 1M while the per-token KV identity did not change. Bigger windows
made the **worst-case KV line item** larger, which is why GQA/MLA/windowed attention
landed in the same years.

Three currencies, often collapsed:

| Currency | What it measures | Lab |
|---|---|---|
| Prefill compute | recompute K/V for tokens you already saw | [prefix reuse](kv-cache-prefix-reuse.md) |
| Resident KV + decode bandwidth | store and re-read T on every new token | this lab |
| Attention quality | softmax budget over those T keys | [context rot](context-rot.md) |
| Occupancy / constraints | which tokens are even allowed in T | [compaction](../sub-apps/sims-context-routing-memory.html#context) |

Prefix reuse does **not** shrink this request’s KV. It skips prefill for a shared
byte-identical prefix across requests. Compaction can shrink T (and therefore KV)
only by deleting or summarizing tokens — P01.

## Evidence base

| Ledger id | Finding | Why it matters for the mental model |
|---|---|---|
| `2305.13245` (GQA) | Share K/V across groups of query heads; interpolation between MHA and MQA | The H_kv term is a design choice; Llama-class GQA (8 vs 64) is an 8× KV cut at the same T |
| `2405.04434` (DeepSeek-V2 / MLA) | Cache a low-rank latent instead of per-head K/V | Further cut of the head axis; still linear in T |
| `2412.19437` (DeepSeek-V3) | Production MLA: ~70 KB/token vs Llama-405B GQA ~516 KB; MTP | Same identity, smaller constant; still linear in T |
| `2502.11089` (NSA) | Train sparse: compressed + selected + sliding-window branches | Cuts T *attended*; T *stored* unless the serving stack offloads |
| `2512.02556` (DSA / V3.2) | Lightning indexer + top-k over MLA latents; core O(Lk) | Indexer itself is still ~O(L²) per layer — next tax |
| `2603.12201` (IndexCache, GLM/Z.ai) | Most DSA layers reuse the nearest full layer’s top-k | Cuts indexer FLOPs (~75%); leaves stored T and core k |
| `2510.26692` (Kimi Linear / KDA) | 3:1 KDA:MLA hybrid; up to 75% less KV, 6× decode at 1M | Linear layers store a state, not T; remaining MLA layers still do |
| `2407.00079` (Mooncake) | KV as first-class: prefill/decode split + DRAM/SSD pool | Cluster-scale *reuse*, not a smaller T on this request |
| `2508.06471` (GLM-4.5) | GQA 96Q/8KV + MTP; deeper-not-wider vs DeepSeek/Kimi | Same GQA rung as Llama; they did not ship MLA |
| `2306.14048` (H2O) | A few “heavy hitter” tokens dominate attention; evict the rest + recency | The T axis can be capped lossily; dropping heavy hitters hurts |
| `2309.17453` (StreamingLLM) | Keep attention-sink (BOS) tokens plus a sliding window | Infinite-stream decode with bounded KV; middle is gone by construction |
| `2608.00101` (Copilot traces) | Intra-turn KV hits ~90% → ~55% across turns; compaction/model-switch invalidate | Production proof the cache is large *and* fragile; growth and reuse interact |
| `2607.29678` (TokTier) | At ~0.99 prefix-hit, tokenization becomes the TTFT bottleneck | When reuse is solved, the next bottleneck moves; KV residency remains |

`2606.17016` (TokenPilot, trend) is the compaction×prefix joint — shrinking T
without mutating the cached prefix. Directional, not a deep-read.

## The dominant direction: cut T, then H, then bytes

Architecture rungs (GQA, MLA, quant) multiply constants. They never cancel the token
axis. Sparse attention (NSA/DSA) multiplies the *attended* T; linear layers (KDA)
replace T with a fixed-size state in most layers. Sliding window / H2O / sinks cap T
by **throwing away the middle** — which is the quality hole in the rot lab. The
harness-side rung that actually reduces T without a custom kernel is **not putting
tokens in the window**: tool bodies to files, traces folded, governance pinned and
small.

## Contested points

- **How far can eviction go?** H2O keeps ~20% heavy-hitters + recent; StreamingLLM
  keeps sinks + window. Both are lossy. Neither is a substitute for pinning
  constraints (`2606.22528`).
- **MLA vs GQA vs MQA** is a hardware/quality Pareto, not a winner. The sim’s
  “MLA-ish” preset is a latent-dim cartoon, not DeepSeek’s exact kernels.
  GLM-4.5 stayed on GQA (`2508.06471`); DeepSeek went MLA then DSA; Kimi mixed
  KDA with MLA (`2510.26692`).
- **Sparse ≠ smaller cache.** NSA/DSA cut *compute* over T. HiSparse-style
  serving still often keeps the full KV so every position stays selectable.
  IndexCache (`2603.12201`) then cuts the *indexer*, not the tensor.
- **mHC (`2512.24880`)** is residual topology (identity mapping). It is not a
  KV lever — do not put it on the Cuts/Leaves grid.
- **Prefix reuse vs residency.** Copilot’s 90→55 cliff is reuse. The 40 GB at 128k
  GQA is residency. Solving one leaves the other.

## Direction of thinking — the transferable moves

1. **Write the identity before the optimization.** If the cost is linear in T, an
   optimization that does not change T, H, bytes, or batch is theater.
2. **A larger advertised window is a larger worst-case working set.** Treat “we
   upgraded to 128k” as a capacity grant that will be filled unless the harness
   refuses.
3. **Decode speed is a bandwidth number at long T**, not a FLOP number. Batching
   more requests is more KV, not free parallelism.
4. **Reuse, residency, quality, and look are different knobs.** Byte-stable
   prefixes, smaller T, where the fact sits in T, and whether softmax even
   sees T ([look vs store](look-vs-store.md)) are four labs for a reason.

## Open questions

- TokenPilot-style compact-without-breaking-prefix (area 09 unpicked; `2606.17016`
  now registered as trend).
- Portable map from LCP / T_stored to billed decode time across providers.
- Whether harness-side T reduction (externalize) beats serving-side eviction on
  constraint survival — joint with P01, unmeasured.

## Try it

Open [the sim](../sub-apps/sim-kv-growth.html): GQA at 8k, drag to 128k, watch KV
cross the weight bar. Then MHA vs GQA at the same T. Then batch 8. Then sliding
window — bandwidth stops growing because T_stored capped; the dropped middle is
the rot lab. Then [look vs store](../sub-apps/sim-look-vs-store.html): DSA
shrinks softmax, not this occupancy bar.