# Learning note — why recall dies as context grows

Companion to [`sub-apps/sim-context-rot.html`](../sub-apps/sim-context-rot.html). KV
growth ([why KV grows](kv-growth.md)) is the serving twin: the same tokens you failed
to drop are the bytes you re-read. This note is the *quality* twin: softmax is a
budget. Ledger IDs below are in `harness-research/ledger/ledger.json`. Chroma’s
context-rot report is the in-window empirical origin (`url:context rot…`, already
read into `papers/01-ctx.md`).

## The question that generates everything

**Is this token in the advertised window, or in the effective one?**

Advertised window = positional encodings and kernels will *accept* N tokens.
Effective window = attention still *uses* the token for the task you care about.
They diverge as N grows. Lost-in-the-Middle (`2307.03172`) is the position half:
U-shaped accuracy, high at start and end, a hole in the middle (~30-point drops
in the original multi-doc QA). Context rot (Chroma, 18 models) is the length half:
performance degrades as input grows even when a needle can still be lexically
found. Maximum effective context (`2509.21361`) names the gap: a 1M model may
have ~40k effective context on multi-hop and hundreds of k on pure retrieval.

```
softmax(q · k_i)  over i = 1..T
more T  →  each key’s mass shrinks
position bias spends leftover mass on the edges
middle keys ≈ invisible  →  “recall went away”
```

Needle-in-a-haystack is the wrong exam for agents. Finding one planted fact is
Mode 1. Chaining two facts, or obeying a constraint from turn 12, is Mode 2 and
dies earlier. Vendors demo Mode 1 at 1M; harnesses fail Mode 2 at 32k.

## Evidence base

| Ledger id | Finding | Why it matters for the mental model |
|---|---|---|
| `2307.03172` (Lost in the Middle) | U-curve: start/end ≫ middle even in long-context models | Position is a first-class reliability variable, not a curiosity |
| `url:context rot…` (Chroma) | 18 LLMs: degradation with input length; NIAH masks reasoning collapse; shuffled haystacks sometimes *better* | Length tax is real; coherent distractors interfere; “just retrieve” is not the agent task |
| `url:effective context engineering…` (Anthropic) | Attention-budget framing; compaction / notes / JIT / sub-agents as the toolbox | Practitioner vocabulary for “don’t fill the window” |
| `2509.21361` (MECW, trend) | Advertised max vs task-specific effective max; gaps up to 99% on hard tasks | The slider in the sim: 128k advertised, ~12k effective on 2-hop |
| `2606.22528` (ConstraintRot) | After compaction, violations 0%→30–59%; 0% if the constraint survives, 38% if dropped | Rot *and* compaction: if the fact is in the hole, summarizing the hole can delete it |
| `2309.17453` (StreamingLLM) | Attention sinks (BOS) + window enable infinite decode | Architecture that *intentionally* drops the middle — bounded KV, bounded recall of the past |
| `2512.02556` (DSA, trend) | Top-k over MLA latents: core O(Lk), indexer still ~O(L²) | Cuts *look*. Unselected tokens usually stay stored; quality of the selected set ≠ advertised window |

## The dominant direction: shrink T, then place what remains at the edges

You cannot “attend harder.” You can (1) put fewer keys in the softmax, (2) put
load-bearing text where mass already is (start = pinned system, end = latest
user), (3) not ask the model to aggregate a haystack it will not see.

That is the same ladder as context management area 01: fold traces, pin
governance, JIT-load bodies. Pinning is not only a P01 safety trick — it is a
**position** trick. Re-injecting a rule in the middle of a 40k dump parks it in
the U-curve hole.

## Contested points

- **NIAH vs rot.** Perfect needle scores are compatible with rotten multi-hop.
  Do not use NIAH as a go/no-go for agent context policy.
- **Newer models still U-shaped.** Long-context training reduced the hole; it
  did not flatten it. Gemini-class 1M still shows middle loss.
- **Shuffled vs coherent haystacks.** Chroma: unstructured sometimes retrieves
  better — structure can *interfere*. Agent traces are highly structured.
  Transfer is not free.
- **DSA vs the U-curve.** Sparse look can skip the middle. It does not flatten
  the curve on the tokens that remain selected, and it does not delete the
  unselected from HBM unless serving offloads. Not a substitute for pin/place.

## Direction of thinking — the transferable moves

1. **Capacity ≠ competence.** Ask “effective for *this* task,” not “what is the
   spec sheet.”
2. **Attention is a budget over T.** Adding tokens taxes every existing token.
   The middle pays first.
3. **Place load-bearing text at the edges, or outside.** Pin at the start;
   latest instruction at the end; haystack in files.
4. **Same T that rots is the T that fills KV.** Quality and serving are one
   occupancy decision. See [kv-growth](kv-growth.md).
5. **Sparse look is not a rot fix.** DSA (`2512.02556`) can skip softmax over
   the middle; those tokens are often still stored, and the U-curve still
   applies to whatever the indexer selected. If the needle is unselected, that
   is a *look* miss, not a thinner budget. Split: [look vs store](look-vs-store.md).

## Open questions

- Read-back vs re-injection under rot (area 01 unpicked): when a ledger
  pointer beats bulk paste as pollution grows.
- Pinning × position × prefix-cache: the pinned block must stay byte-stable
  *and* at the high-recall edge (joint with P01/P17).
- Task-conditioned MECW from our own journals — nobody publishes a per-task
  effective-window for coding agents.

## Try it

Open [the sim](../sub-apps/sim-context-rot.html): fact at 50%, grow 8k→64k, then
flip Needle → 2-hop. Then at 32k slide the fact start→middle→end. The U is the
whole argument. Compaction’s job is to stop the haystack from existing; pinning
is to keep the rule off the U’s trough.