# Learning note — looking is not storing Companion to [`sub-apps/sim-look-vs-store.html`](../sub-apps/sim-look-vs-store.html). The [growth lab](kv-growth.md) asks what a token costs to carry. This lab splits that cost: **softmax over T** vs **bytes of T resident**. Every arXiv ID is in `harness-research/ledger/ledger.json`. ## The question that generates everything **If decode got cheaper, why is the GPU still full?** Because “attention got cheaper” names three different taxes: ``` look = how many past tokens enter this step’s softmax / top-k store = how many past tokens still occupy HBM (selectable next step) thin = bytes per stored token (GQA / MLA / quant) ``` Sparse methods (NSA, DSA) cut *look*. Latent methods (MLA) cut *thin*. Linear hybrids (KDA) cut *store* in most layers. Serving pools (Mooncake) reuse store across requests. Mixing the four in one sentence is how the identity gets lost. ## Background — three labs, three bets (2024–26) **DeepSeek** started at the *thin* axis. Multi-head Latent Attention (`2405.04434`, production numbers in `2412.19437`) caches a low-rank joint latent plus a small RoPE component instead of per-head K/V. DeepSeek-V3 reports ~70 KB/token vs Qwen-2.5-72B GQA ~328 KB and Llama-3.1-405B GQA ~516 KB. T is still linear; every stored token is still *looked at*. Native Sparse Attention (`2502.11089`) then trains three branches — compressed, selected, sliding window — so pretraining never pays dense O(T²). DeepSeek Sparse Attention (`2512.02556`) sits on MLA: a lightning indexer scores the prefix and the main softmax runs on top-k latents. Core attention becomes O(Lk). The indexer itself remains roughly O(L²) per layer, and production stacks often keep the full KV so every position stays selectable. mHC (`2512.24880`) is residual topology (identity mapping). It is not a KV lever. **GLM / Z.ai** stayed on *thin-via-GQA* in 4.5 (`2508.06471`): 96 query heads, 8 KV heads, partial RoPE, QK-Norm, an MTP head for speculative decode, and a deeper-not-wider MoE versus DeepSeek/Kimi. They did not ship MLA. GLM-5 (`2602.15763`) then *imports* DSA. IndexCache (`2603.12201`) is the GLM-side original: adjacent DSA layers pick highly overlapping top-k (reported 70–100% overlap), so most layers reuse the nearest full-layer index. On a 30B DSA model at 200k: drop 75% of indexer compute, 1.82× prefill / 1.48× decode, quality essentially flat. IndexShare in later GLM releases is the same reuse idea in production. IndexCache cuts the *indexer*, not stored T. **Kimi / Moonshot** split across serving and architecture. Mooncake (`2407.00079`) treats KV as the first-class object: prefill/decode disaggregation plus a DRAM/SSD pool, prediction-based rejection under overload. Simulated throughput up to +525% while holding SLOs; real Kimi traffic +75% more requests. That is cluster-scale *reuse*, not a smaller T on this request (the prefix-reuse lab). Kimi Linear (`2510.26692`) is the occupancy bet: Kimi Delta Attention (channel-wise gated delta rule) mixed 3:1 with MLA. Reported: up to 75% less KV than full MLA, 3.98× on RULER 128k, 6.3× decode at 1M vs MLA. The remaining MLA layers still store T. K3 later stacks KDA with Attention Residuals; the Linear paper is the mechanism you can actually inspect. ## Evidence base | Ledger id | Finding | Axis | |---|---|---| | `2405.04434` | MLA: cache a latent, not per-head K/V | thin | | `2412.19437` | ~70 KB/token MLA vs ~516 KB Llama-405B GQA | thin, production constant | | `2502.11089` | NSA trains sparse (compress + select + window) | look | | `2512.02556` | DSA: indexer + top-k on MLA; core O(Lk) | look; indexer still quadratic | | `2508.06471` | GLM-4.5: GQA 96Q/8KV + MTP, not MLA | thin (GQA rung) | | `2602.15763` | GLM-5 adopts DSA | look, imported | | `2603.12201` | IndexCache: 75% of indexers reused; 1.82× / 1.48× | indexer, not store | | `2510.26692` | KDA 3:1: ≤75% less KV, 6.3× decode at 1M vs MLA | store (¾ of layers) | | `2407.00079` | Mooncake: KV-centric disagg; up to 525% throughput | reuse, not T | | `2512.24880` | mHC: manifold-constrained residuals | not this lab | Numbers are the papers’ own. The sim is a cartoon of *direction*: DSA leaves store=T, KDA sets store≈T/4, IndexCache leaves both look and store. ## Contested points - **Does DSA shrink the cache?** Only if serving offloads unselected KV. Default DSA is a compute win with occupancy still O(T). Do not read “sparse” as “smaller HBM.” - **MLA vs GQA** is a Pareto, not a ranking. GLM-4.5 spent the budget on depth and MTP instead. DeepSeek then stacked DSA on MLA. Kimi mixed linear layers *with* MLA rather than replacing it. - **Indexer cost is the next tax after DSA.** IndexCache / IndexShare attack that tax. They do not change k or T. - **KDA quality at frontier scale** is evidenced at 48B-total / 3B-active (`2510.26692`). K3’s 3T-class claims inherit the mechanism; treat the 6.3× as Linear-paper, not as a K3 audit. - **Mooncake vs prefix reuse.** Same currency (skip recompute). Different substrate (cluster DRAM/SSD vs GPU prefix cache). Neither shortens this request’s bar. ## Direction of thinking — the transferable moves 1. **Split the sentence “attention is expensive” into look / store / thin / reuse before picking a paper.** If the OOM is occupancy, DSA will disappoint; if the bill is prefill, Mooncake/prefix will not. 2. **Sparse by default keeps the unselected tokens.** Offload is an extra serving decision. Feel it in the sim: teal follows yellow only on “DSA + offload.” 3. **A hybrid still has a dense remainder.** KDA 3:1 stores T on the MLA layers. The 75% cut is the other three quarters, not magic on the last. 4. **Harness-side still cuts T by not putting tokens in the window.** None of these kernels replace pin / compact / externalize. ## Open questions - Joint: DSA look-cut × TokenPilot compact-without-breaking-prefix (`2606.17016`) — unmeasured. - Whether indexer-reuse (IndexCache) and KV-offload compose, or whether shared indices across layers fight layer-wise prefetch. - Portable map from (look, store, thin) to billed decode across providers that hide the kernel. ## Try it Open [the sim](../sub-apps/sim-look-vs-store.html): T=32k, click DSA — yellow collapses, teal does not. Then KDA — teal drops. Then IndexCache — only the indexer stat moves. Then “DSA + offload” so you can feel the serving extra that the papers do not assume.