feat(moe): add DEGRADE_ZERO miss-slot zero-fill policy (issue #865)

[kritikagarg]

Introduce DEGRADE_ZERO=1 / DEGRADE_TAU= (default 0.03): when a prefetch deadline is missed, experts whose aggregate gate weight across the batch is below tau are zero-filled instead of triggering a blocking demand-load. Issue #865 measures tau=0.03 zeroes 21.8% of slots for only +2.9% perplexity — well within acceptable quality budget on NVMe-bound workloads where the stall cost dominates.

Design constraints respected:

• Decode-only (S<=4 guard): same invariant as EXPERT_BUDGET — dropping experts during prefill corrupts the KV cache (Diagnostic: token-exactness sweep + performance breakdown (pipe2 S=1 resident vs CPU vs CUDA — 4 prompts x 5 configs) #292)
• Miss-only: pin/LRU hits are never dropped; they load for free
• Rescue rule: no position is left with zero routed experts
• Renormalises surviving weights for both norm_topk=1 (GLM-5.2) and norm_topk=0 (other MoE configs) to preserve output magnitude
• Opt-in only (default OFF): output changes, must never be silent
• Stats footer reports tau and cumulative zeroed slot count when active

Adds tests/test_degrade_zero.c: 8 properties, 20 checks, standalone (no model or weights needed), auto-discovered by the Makefile test gate.

Summary

Describe the problem and the smallest change that solves it.

Validation

• make -C c check
• CUDA changes were tested with make -C c cuda-test (if applicable)
• Performance claims include hardware, commands, and repeatable measurements

Compatibility

• The default CPU build remains dependency-free
• No model files, generated binaries, or benchmark artifacts are included

[kritikagarg]

Updated the implementation to address the two semantic points raised in #865:

1. Per-position τ gate (was: aggregate)
The original code summed gate weight across all batch positions before
comparing to τ. This coupled positions — at S>1 a slot could survive because
other positions in the batch pushed the aggregate over the threshold, even
if every individual position was sub-τ. The gate now tests each position's
own weight independently. At S=1 (the common decode case) the two are
identical; at S=2–4 the per-position gate is stricter and matches the
measurement methodology the +2.9% ppl number was taken under.

2. No renorm (was: renorm after drop)
The previous apply step renormalised surviving weights — either to
sum-1×routed_scale (norm_topk=1) or by old_sum/new_sum (norm_topk=0). Both
paths redistribute the dropped mass upward rather than losing it, which is a
different approximation with no measured quality curve. Survivors now keep
their original weights unchanged. The approximation is the dropped mass.

Tests: replaced P4 (renorm-normtopk) and P5 (renorm-nonnorm) with P4
(no-renorm: survivor weight is byte-identical to input) and P5
(per-position-tau: S=2 shared expert kept when one position meets τ, dropped
when all are below). All 20 checks pass.

[JustVugg]

Thank you for taking this — @outtodata's #865 is one of the better-measured proposals in the tracker and it deserved an implementation.

I want to read it properly rather than fold it into a batch, because it is the first change here that makes the engine produce different tokens on purpose. Everything else in this repo is either exact or explicitly a quantisation choice made at conversion time; this is a runtime decision to skip work and accept the output moving. That is a legitimate capability and #865's numbers support it — zeroing slots below τ=0.03 costs +2.9% ppl on OLMoE while τ=0.05 costs +41%, so the operating point is real and narrow.

Three things I will be looking for, said up front so you can point me at them or fix them before I do:

Off by default, and loud when on. A run that silently approximates is indistinguishable from a run that is subtly broken, and six months later someone files a quality bug we cannot reproduce. The [PROF] line should report how many slots were zeroed, not just that the flag was set.

The gate on weight, not on count. #865 is explicit that the weight threshold is the control and the slot count is not: τ=0.03 zeroes 21.8% of slots for +2.9%, but "zero 21.8% of slots" chosen by any other rule would not give that number.

A token-exactness assertion with the flag off. The existing oracles must stay bit-identical, so that enabling this is the only way to change output.

+473 lines across 3 files and CI has not reported yet — I have approved the workflow run. I will come back to it with a real review rather than a checklist.

One thing worth flagging because it affects where this pays off: #865's measurements are on OLMoE, and @outtodata says so plainly. This engine's own profile shows expert I/O dominating expert matmul by roughly 5:1 on a streaming box, so a policy that avoids waiting for a late read is aimed at exactly the right cost — but the ppl numbers that justify τ=0.03 are one model's. Worth stating in the docs that the default is calibrated on OLMoE, not measured on GLM or Kimi.

[outtodata]

Thanks for reading it this way — the three gates you named are exactly the ones the measurements support. Tying each back to the data, plus two places where an implementation can silently drift from them:

Off by default, loud when on. Agreed, and the [PROF] counter is mandatory from our side too. One refinement worth asking for: report the zeroed share per layer, not just a global count. τ is a global knob but the drop share is not uniform across layers — a global "21.8% zeroed" can hide one layer zeroing far more, and that is where the ppl cost concentrates. A per-layer min/max next to the hit/miss line makes a silently-miscalibrated run visible instead of indistinguishable from a healthy one. I can pull the per-layer drop-share table from our grid run if that helps the review.

Gate on weight, not on count. This is the load-bearing point, and I'm glad it's called out up front. The +2.9% at τ=0.03 comes from the weight threshold selecting which slots to drop — the slots contributing least to the MoE output under norm_topk_prob=False. A count-based rule ("drop 21.8% of slots, chosen any other way") selects a different set, and there is no reason to expect it lands anywhere near +2.9%. Related, from our renorm A/B on the same harness: the reason zeroing is cheap at all is that OLMoE's router contract lets the top-k weights sum to less than 1, so dropping a slot only ever removes a small amount of mass — a bounded, subtractive deviation. Renormalizing survivors instead was catastrophically worse at every τ (+6253% ppl at τ=0.03). Keep the flag's semantics exactly "skip slots with gate weight < τ"; any rescaling of survivors changes the regime, not the degree.

Token-exactness with the flag off. Should hold naturally: the gate sits before pipe_dispatch / the blocking load, so with the flag off the path is untouched and the existing oracles stay bit-identical. The one place to look hard is the PIPE path: a zeroed slot must still produce a well-defined QT view (or be excluded from the matmul consumer list) so the pipelined consumer never reads a half-loaded expert. The serial path is trivial; slot lifetime under PIPE=1 is where this breaks subtly.

OLMoE-calibrated default. Fair, and worth stating in the docs exactly as you put it. For the record, the numbers behind τ=0.03: OLMoE-1B-7B, fp16 offload, ppl 13.9144 → 14.3194 (+2.9%), task_acc unchanged at 0.889, ~22% of slots dropped; τ=0.05 → +41% ppl, so the operating point is real and narrow. GLM (norm_topk=1) and Kimi have different router contracts and expert counts — until someone measures them, the default should ship documented as calibrated on OLMoE, with per-model τ left to the operator. Happy to re-run any τ point or ablation if you want one more data point before merge.