Looking is not storing
Sparse attention can stop looking at a token without stopping storing it. Linear layers do the reverse in most of the stack. Decode getting cheaper does not mean the GPU got emptier.
You bought a 128k window. The vendor shipped “sparse attention.” Time-per-output-token dropped. Batch-4 still OOMs. The growth lab asked what a token costs to carry. This lab asks which bill the architecture actually paid.
Architecture
stored (resident KV)
looked at this decode
Looked at
—
Stored tokens
—
Resident KV
—
Indexer work
—
Two bills — yellow is compute, teal is occupancy
Cuts vs leaves — same identity, different axes
| Bet | Cuts | Leaves | Who |
|---|---|---|---|
| GQA | H (8 KV heads vs 96 Q) | T still linear | GLM-4.5 2508.06471 |
| MLA | H further (latent ~70 KB/tok vs GQA hundreds) | T still linear; still look at all T | DeepSeek V2/V3 2405.04434 |
| NSA / DSA | T looked at (top-k / branches) | T stored, unless serving offloads | DeepSeek 2502.11089 · 2512.02556 |
| IndexCache | Indexer FLOPs (~75% of layers reuse top-k) | core k and stored T | Z.ai / THU 2603.12201 |
| KDA 3:1 | T in 3/4 of layers (finite-state) | MLA layers still store T | Kimi Linear 2510.26692 |
| Mooncake | Prefill recompute across the cluster | This request’s T | Kimi serving 2407.00079 |
mHC (2512.24880) is residual topology. It is not on this grid. Offload is a serving choice, not what DSA ships by default.
What to try
- Leave T at 32k. Click DSA. Yellow shrinks to k. Teal does not move. That is the OOM-with-faster-decode failure.
- Now KDA 3:1. Teal drops to the MLA quarter. Kimi paid the occupancy bill; DeepSeek paid the look bill.
- Click + IndexCache after DSA. Look and store stay put; only indexer work falls. Next tax after DSA, not a smaller cache.
- DSA + offload makes teal follow yellow. Feel it — then remember this is extra serving work, not the paper’s default.
- Drag T to 128k on DSA, then on GQA. GQA look and store both explode; DSA only the store bar does.
Ask first: did we stop looking, stop storing, or only thin each stored token? Sparse ≠ smaller KV. Linear ≠ free quality on the MLA layers. Cluster reuse (Mooncake) is a fourth bill, already in the prefix-reuse lab.