# Learning note — model routing & cost

Companion to [`sub-apps/sims-context-routing-memory.html`](../sub-apps/sims-context-routing-memory.html) (routing widget). Evidence and
reasoning path here; reflexes in the sim. Every arXiv ID is registered in `harness-research/ledger/ledger.json`.

## The question that generates everything

**What fraction of my traffic genuinely needs frontier capability?**

Not "which model is best" — that question has one answer and one bill. The routing question is a *population*
question: the traffic splits into a slice that exhausts a cheap model's capability and a slice that doesn't, and
the entire economics is decided by the size of that slice. The sim's third slider is the field's central
unknown, and everything else — savings, quality blend, break-even — is arithmetic on it.

```
                ┌─ hard slice (X%)  ──▶ Frontier  $15/$75 per 1M ── quality 95
traffic ──router┤
                └─ easy slice (100−X%) ─▶ Fast     $0.25/$1.25 ─── quality 65
```

## The arithmetic of the 60× gap

Cost per request = tokens ÷ 1M × (input$ × 0.6 + output$ × 0.4) — the 0.6/0.4 split encodes the empirical
fact that generation dominates billing for chat-shaped workloads. At the sim's defaults (2K tokens/request):

| Tier | Blend $/1M | Per request | Per 1,000 requests/day | Per month |
|---|---|---|---|---|
| Frontier | $39.00 | $0.0780 | $78.00 | $2,340 |
| Mid | $7.80 | $0.0156 | $15.60 | $468 |
| Fast | $0.65 | $0.0013 | $1.30 | $39 |

Frontier-to-Fast is a **60× price ratio**. That ratio is what makes routing worth a router: every percentage
point of traffic moved off the frontier scales that slice's cost by ~1/60, so total savings grow almost
linearly as the hard share shrinks. At 20% hard: blended cost = 0.2×39 + 0.8×0.65 = $8.32 per 1M-equivalent →
**78.7% savings** at blended quality (0.2×95 + 0.8×65) = **71%**. Note what the linear blend assumes: easy
tasks get *zero* quality benefit from the frontier. That is pessimistic by construction — real "easy" tasks
often see no measurable drop — which is why measured routing beats the worst case. At 10K requests/day the
same defaults put All-Frontier at $23,400/month and the router at $4,992 — the absolute numbers, not the
percentages, are what make routing an organizational decision.

Break-even intuition: savings ≈ (100 − hard%) × (1 − 1/60) ≈ 98.3% of the easy share. Routing buys little only
when the hard share approaches ~100% — i.e., when you can no longer honestly classify anything as easy. The
failure mode is not the arithmetic; it is *mislabeled traffic*.

## Evidence base

| Ledger id | Finding | What it changes in the model |
|---|---|---|
| `2607.22465` (TRACE-ROUTER) | Route once per task at admission via contextual bandit, pin all calls, train on terminal reward: Terminal-Bench +7.1 accuracy pts at −36% latency; +7–8 pts on tau2-Bench | Routing has a correct *granularity*: the unit of supervision. Per-call routing can't be trained (reward arrives at task end) and breaks prefix-cache continuity |
| `2608.06867` (LLMRouter) | xRouteBench: routing as sequential decision over 16 routers; learned routers beat the best fixed model by 14.6% relative | "Pick one model for everything" is measurably dominated — when routing is done as a learned, benchmarked decision layer |
| `2608.20316` (Pandora's Box) | Routing's own estimation is *costly inspection*; the optimal policy matches exhaustive quality with far fewer estimator queries | The router has an overhead budget too — meta-cost is a first-class term, not a footnote |
| `2606.22840` (RLM-Cascade) | Proxy-layer response-level speculation: easy turns to a draft model, −45.8% cost at 1.83× p50 on Claude Code traffic — n=125, single 20-task check | The cascade pattern works but the evidence is thin; also introduced a schema-critical-turn bypass *after* the fact — misclassification has a face |
| `2608.15127` (AgentSysBench) | Workload ground truth: task-aware serving −29–40% latency; tool-result caching removes 35.2% redundant searches | Measure the workload first; the routing win is one of several measured levers, not a default |
| `2608.00101` (Copilot traces) | Model switches drastically invalidate KV reuse | Routing has a *cache externality*: every tier flip is also a cache invalidation event |
| `2608.19677` (CacheRoute) | Affinity routing 64→93% hit rate, with published failure workloads | Placement and routing are the same problem seen from two layers; both need measurement gates |

## The dominant direction: route at the unit of supervision, pin for affinity

TRACE-ROUTER's structural finding generalizes beyond its benchmark: supervision (did the task succeed? at
what cost/latency?) arrives once per task, so the routing decision should be made once per task and *pinned* —
fixing both the credit-assignment bug of per-call routers and the cache physics (one backend holds the
prefix; see the KV note for why that matters). The unresolved cost of pinning is **escalation asymmetry**: a
task admitted to a cheap tier cannot upgrade when it turns hard, so admission mistakes are paid for the whole
task. Difficulty classification, not price arithmetic, is the hard problem — the sim's slider is honest about
being a slider and not a classifier.

## Contested points

- **Fine-and-learned vs coarse-and-pinned.** LLMRouter benchmarks sequential routing decisions (learn more,
  route more finely); TRACE-ROUTER pins one decision per task (route coarser, learn on terminal reward). Not
  flatly contradictory — sequential ≠ independent per-call — but unresolved at benchmark level.
- **Does routing pay after its own overhead?** Pandora's Box formalizes estimation as costly inspection: a
  router that queries an expensive quality estimator per request can eat its own savings. Deployed routers
  gate on cheap signals (heuristics, small classifiers, historical per-task stats).
- **What counts as "easy"?** RLM-Cascade's post-hoc bypass for schema-critical turns is the honest admission:
  the easy/hard boundary is workload-specific, discovered by measurement, and misclassification concentrates
  quality loss exactly on the turns you were sure about.

## Direction of thinking — the transferable moves

1. **Split the population by what the expensive resource actually requires.** The question is never "can the
   cheap model do this task?" but "what does this task *consume* that only the expensive model provides?"
2. **Pay up only for the qualifying slice — and measure the split instead of guessing it.** The sim's slider
   is a stand-in for a classifier you should build from recorded outcomes.
3. **Count the router's own cost.** Estimation queries, misclassification losses, cache invalidations, and
   escalation asymmetry are all real line items (Pandora, Copilot, TRACE-ROUTER respectively).
4. **Match decision granularity to supervision granularity.** If you can only measure success at task end,
   make routing decisions at task start and hold them.

## Open questions

- Complexity-gated tier routing of trivial turns remains a hypothesis: RLM-Cascade's evidence is n=125 with a
  single 20-task check (area 09, proposal P2 #8 — test on recorded sessions via replay before adopting).
- Bandit-trained admission needs terminal rewards, which arrive slowly; no window paper demonstrates
  online learning of admission policies with mid-task escalation as a safe fallback.
- Routing × cache affinity co-optimization: TRACE-ROUTER pins for supervision, CacheRoute plans for affinity;
  no paper optimizes both objectives in one placement layer.

## Try it

Open [the sim](../sub-apps/sims-context-routing-memory.html): set the sliders to your own traffic shape and read
the Smart Router row first — savings vs quality move in opposite directions as the hard share grows. Then
find *your* break-even: the hard-share value at which the router's edge stops justifying a classifier. Watch
the bars, not just the table — the frontier bar's dominance at volume is the whole argument.