Proposal: a self-sizing (dynamic) expert cache

[Helldez]

Proposal only — no implementation. The diff is one roadmap entry; this description is the
actual content. Opening it to settle the intent before writing any code.

Intent: the expert cache should size itself

The cache budget is a static decision today. --cache-mb auto reads available RAM once at load
and holds that number for the whole run; every other path is a constant somebody picked by hand.
After load, neither the model nor the device has any say. All three of those are wrong in ways that
are visible in data already committed here.

The same number means different things per model. The budget that matters is not MiB, it is how
many token cycles it holds — a cycle being what one token routes across every layer,
TD = k × Σ entry_bytes(il). At a fixed 3000 MiB:

model	k	TD (MiB)	cycles held
gpt-oss-120b	4	1815	1.65
Qwen3-30B-A3B	8	1046	2.87
Qwen3.6-35B	8	581	5.16
Gemma-4-26B	6	676	4.44
LFM2.5-8B	4	654	4.59

Three times the cache, same setting. TD is arithmetic over the gguf and is known at bind — predicted
1814 against a measured token_demand_MiB=1815.4 on gpt-oss — and nothing currently uses it.

The same number means different things per device, and over time. A budget that is healthy on
one phone loses the dense weights on another, and the same run degrades as the device warms: at a
fixed 3000 MiB, one committed 1350-token run goes 4.52 → 2.17 tok/s while faults rise 20×. A value
chosen once at load cannot see any of that.

And it is sized from a signal that does not predict the cost. Across the 38 committed runs with
per-token telemetry, the rank correlation between mem_available_mib and majflt swings from
−0.49 to +0.64 depending on the run. Two other signals do track it — dense_resident_frac and
swap_mib:

run	dres	swap	tok/s
qwen3.6 c3000 ahwb	1.000	294	3.066
qwen3.6 c3000 anon	0.848	563	2.591
gpt-oss c2000 lanes 4	0.776	638	0.996
gpt-oss c2000 lanes 8	0.923	210	1.300

Within a run, faults against tok/s on 100-token windows give −0.71 / −0.62 / −0.57 on the three
long runs.

What the routing traces contribute

Replaying the committed traces (docs/bench-data/2026-07-15-route-trace/) through
scripts/route-replay.py, with budgets set to exact multiples of TD:

cycles held	gpt-oss-120b k2	Qwen3-30B k6	Gemma-4-26B k6
0.5	0.0	0.0	0.0
1	17.6	37.7	27.9
2	30.6	47.6	47.3
4	50.1	71.8	70.2
6	57.7	86.3	82.2
8	63.2	92.9	91.0

Two things a dynamic policy can build on. The cliff below one cycle is exactly 0.0 % on all three
models, offline — it is LRU against the deterministic layer visit order, not a device artifact —
so it is a hard boundary, not a tunable. And there is no knee: marginal gain decays smoothly
(gpt-oss +17.6, +13.0, +11.5, +8.0, +4.0, …), so any "stop at N cycles" rule would be an invented
constant. An earlier draft of this proposal used 4 cycles; on Qwen that would have left 15 points
of hit rate on the table.

A potential algorithm

Sketch, not a specification. The point is that every term is either derived or measured, so no
constant is tuned per model or per device.

TD = Σ_il k · entry_bytes(il)                    // at bind, from shapes alone
B0 = TD                                          // one cycle: the best purchase there is
legal budgets: {0} ∪ [TD, ∞), in whole multiples of TD

during prefill:
    Δhit(n) simulated from the routing counters   // LRU over counters, no I/O

between generations (never during a decode):
    benefit = Δhit(n→n+1) · TD / bandwidth        // bandwidth = read_bytes/io_ms, this turn
    cost    = Δ(faults · ms_per_fault + compute)  // measured by having grown
    grow while benefit > cost

Constraints it is built to respect:

• Quality is never touched. Only the budget moves, and a budget change is byte-identical by
  construction (gate S3). k and drop_cold_frac change the output and stay exactly where the user
  put them for the whole session.
• Nothing moves inside a generation. set_cache_budget() is a between-turns operation and the
  proposal keeps it that way. This is a real limitation: intra-run degradation is seen only at the
  next turn.
• No branch per platform. On desktop the device concedes the RAM and growth continues; on a
  phone it does not and growth stops. Same code.

Why this is not the retired governor

--cache-dynamic (gov2) was measured a net loss and removed (docs/pressure.md). Two differences:
it chased MemAvailable, the signal shown above to be noise; and it resized continuously, so a
shrink could land inside the cliff, where you pay all the RAM and all the management for a 0 % hit
rate. Quantising to whole cycles — with the last step down going to zero rather than to TD−ε —
removes that failure mode.

Open questions

1. Is the dynamic part worth it at all? In every committed run where the cache produces hits it
   wins by 20–30× on the ms-saved / ms-lost balance; the only zeros are budgets below one cycle.
   It is entirely possible that a correct floor captures nearly all the available win and the growth
   loop is not worth its complexity. That would be the good outcome.
2. The cost side is under-counted. A linear ms-per-fault fit attributes 11.5 % of wall to faults
   in the 1395-faults/token run, but that run does 1.877 tok/s against 3.066 for the same
   configuration when clean — a 5–10× larger real penalty. Reclaim hurts beyond the faults it counts.
3. Desktop cannot measure the cost today. major_faults() returns 0 and dense_resident_frac
   is −1 on Windows by design (core/src/io/platform_io.cpp:126-130). Worth noting auto picked
   budgets between 5280 and 7420 MiB for the same model on the same machine in one afternoon, and
   that spread confounded the lane comparison in the desktop campaign.
4. Missing traces. Only three models have committed routing traces; Qwen3.6 and LFM2.5 would
   need a short --route-trace run each before their curves can be checked against the rest.

If it goes ahead

Three separable pieces, increasing risk:

1. TD at bind plus the floor {0} ∪ [TD, ∞), replacing cache_min_mb. Closes the cliff, which is
   how the current default fails silently. Useful on its own, and static.
2. The prefill-counter curve simulator (a C++ port of logic already validated in
   scripts/route-replay.py).
3. The between-turns growth loop — the genuinely dynamic part, and the one that needs a long-run
   A/B before anyone believes it.