feat(ds4): add DeepSeek-V4-Flash-0731 — routed-expert paging, +28.8% cache budget, and an experimental prefetch adapter

[nwoolmer]

What

Adds DeepSeek-V4-Flash-0731 end to end: the model in the registry, the routed-expert
paging that lets it run when the experts do not fit, and an experimental
expert-prediction adapter for speculative prefetch.

The paging is the shippable part. The adapter is published as a measured negative
result — it is off by default and, as implemented, a regression. Details below rather
than buried.

Measured on starling (gfx1151, Radeon 8060S), deepseek-v4-flash-0731.mq2lloyd.

---

1. The model

deepseek-v4-flash-0731 → hipfire-models/hipfire-deepseek-v4-flash-0731, arch_id=9,
43 layers, 256 routed experts + 1 shared, top-6. Alias deepseek4:0731.

Ships the 3-stage DSpark draft chain instead of the classic nextn MTP head, so it
pairs with a -dspark sidecar, not a -mtp one. temp=1.0 mirrors the base
checkpoint's serving guidance.

registry_gen.py's arch_id_for() matched deepseek-v4-flash exactly and failed
closed on the dated tag ("no arch_id mapping"). Widened to a prefix match so future
dated checkpoints do not each need an arm — the fail-closed behaviour was correct and
is preserved.

Model card: docs/models/deepseek-v4-flash-0731.md.

---

2. Routed-expert paging

Per-layer expert blobs become bounded slot pools; experts are pread from the HFQ
on demand and the device pointer table is repointed before each MoE dispatch.

Default OFF. With HIPFIRE_DEEPSEEK4_EXPERT_CACHE_GB unset the loader uploads all
n_routed_experts and the forward path is the code it was. An 8 GiB expert cache
replaces 72.2 GiB of resident experts, byte-identical output.

EXPERT_CACHE_GB=auto — the main performance dial (+28.8%)

budget	slots/blob	hit rate	tok/s	ms/token	expert I/O share
4 GiB	14	40.5%	7.02	142	—
8 GiB	28	59.4%	7.06	141.7	41.9%
16 GiB	56	74.4%	—	—	—
32 GiB	112	86.5%	9.09	110.0	26.5%
64 GiB	224	91.0%	—	—	—
resident	256	91.3%	—	—	—

auto takes whatever fits after non-expert weights, KV and headroom, clamped against
MemAvailable; if everything fits it disables paging rather than paging pointlessly.
Hit rate saturates at 91.3% (rest is compulsory cold-miss), so past ~32 GiB the curve
flattens — 64 GiB buys another 4.5pp.

Picking the number by hand costs real throughput and fails silently, which is why
auto exists. Guarded: if /proc/meminfo is unreadable the sentinel would have sized
a pool from u64::MAX, so that now errors with an actionable message.

Where decode time actually goes

Ablating routed MoE (8 GiB budget):

component	ms/token	share
expert paging I/O	59.4	42%
routed expert GEMV compute	14.6	10%
attention, shared experts, norms, lm_head	67.7	48%

The routed-MoE path costs 74 ms/token and 80% of that is I/O, not compute — an
I/O-bound workload wearing a compute-bound costume. This is why cache size dominates
and why hiding latency does not.

Correctness

scripts/paging-neutrality-gate.sh compares committed token IDs (not text — BPE can
render two different token sequences identically) across three arms: fully resident, a
cache large enough for what the prompt touches, and one small enough to thrash. All
byte-identical over 192 greedy tokens. PAGING_GATE_FAST=1 runs it in 45 s instead of
50 min by restricting routed experts to the first N layers.

Also fixes a pre-existing HIP-graph correctness bug found by the gate: the APE path
baked a host-resolved pointer at capture time, so graph replay diverged from direct
dispatch on every ds4 run, paged or not.

---

3. Expert-prediction adapter (experimental — OFF by default)

deepseek-v4-flash-0731-adapter-r128.bin, 44.6 MB. Predicts layer L+1's expert
selection from layer L's hidden state so the pager can fetch a layer early:

z_{L+1} ~= B · (A · h_L) + gate_bias_{L+1}     A: [128,4096]  B: [256,128]

The frozen native router still makes the real selection — the adapter only ranks
prefetch candidates, so a wrong prediction costs a wasted fetch, never a wrong token.
Output byte-identical with it on or off (greedy, 9/9 arms).

Listing it in the registry does not enable it; the runtime loads one only when
HIPFIRE_DEEPSEEK4_EXPERT_ADAPTER points at a file.

It works as a predictor

ExpertRecall@M — of the 6 experts actually chosen, how many appear in the top-M:

top_M	recall	covers of 6	wasted	fetched late	fetches vs baseline
4	59.6%	3.58	0.42	2.42	1.07x
6	75.9%	4.55	1.45	1.45	1.24x
8	83.2%	4.99	3.01	1.01	1.50x
12	89.2%	5.35	6.65	0.65	2.11x

Training-free alternatives were measured and rejected first — recency covers 0.1%
of the miss stream, a token-conditioned profile 13.0% (nearly all of it the 3
hash-routed layers we already stage exactly), and running the real gate_{L+1} on
h_L 4.2% (27.6% recall). ds4's mHC replaces the residual stream with 4
Sinkhorn-mixed streams, so there is no slowly-evolving residual to read through.

Rank matters more here than in published work: SpecPrefetch reports ~84% at r=32 on
64-expert DeepSeek-VL2; ds4 is 256-choose-6 and manages 54.9% at r=32, 75.6% at r=128.

It is still a regression

cache	best top_M	speedup
4 GiB	3	0.89x
8 GiB	4	0.93x
32 GiB	6	0.86x

Cause, measured directly on the same workload:

off:  70,736 accesses,  25,218 misses,   83.1 GiB read
on : 147,200 accesses,  52,900 misses,  174.4 GiB read   (2.10x)

Speculation more than doubles bytes read. Hit rate stays flat (~64%) while bytes
double — churn, not caching: staged experts evict entries the real dispatch then needs.
No amount of overlap pays for 2.1x the I/O.

Tightening the cache made it worse, so "constrained caches are where prefetch pays"
is refuted, not merely untested. Prefetch overhead is a fixed per-layer cost while its
benefit scales with how much I/O there is to hide, so it competes with the cache
budget rather than composing with it.

Supporting infrastructure (useful independently)

• Async transport: hipHostMalloc/hipHostFree bound, memcpy_htod_offset_async
  added, pinned staging + dedicated copy stream + event wait, and the pread moved to
  a worker thread. Verified byte-exact on out-of-order ranges with staging reuse
  (examples/async_prefetch_smoke.rs). Pinning is load-bearing, not an optimisation —
  hipMemcpyAsync from pageable memory silently degrades to a synchronous
  bounce-buffer copy.
• Instrumentation: token-tagged expert trace, route-lookahead diagnostic, and a
  probe showing per-dispatch D2H is 98% pipeline drain (6.1 µs idle vs 317.8 µs
  with work queued).

Where it might still pay (untested)

• Discrete GPUs (R9700 gfx1201, 7900 XTX gfx1100) — on unified memory the H2D half
  is a plain memcpy with nothing to hide; on a discrete card it is a real PCIe DMA.
• Removing the per-dispatch host round-trip — the pager is host-driven, so every
  dispatch copies routing back to the host, including the ~64% that are pure hits
  needing no host action.

---

Compatibility

• EP sharding: refused with a clear error (the shard path aims non-owned experts at
  a zeroed dummy, which paging must never leave in place).
• TP expert slicing: refused pending expert_tp_row_gather from [Superseded] refactor: unify device mesh, model parallelism, and generation dispatch #527. Every buffer
  on the fill path is already sized for it (ExpertFillTransform, exercised by
  synthetic slicing tests); only the concrete gather is missing. A 3-step removal recipe
  is in the guard.

Risk

Default-off for both paging and the adapter. The one change on the always-on path is
the HIP-graph APE fix, which corrects a real pre-existing bug and was verified
bit-identical over 96 steps.

[fivetide]

would you check if it would be doable ontop of #527 ? that one does a lot of changes into weight loading and might be fully incompatible otherwise..

[nwoolmer]

Compatibility seams for #527 (device mesh / model parallelism)

#527 unifies device mesh, TP and EP, and touches all seven of the files this PR does. I
checked the overlap and landed the seams here rather than leaving them for merge time.
#527 needs no changes — the only shared surface is a pub fn it already exports.
Integration notes are posted there: #527 (comment).

Two seams are in, both inert today and both proven by negative control (neutering each one
makes its tests fail, so they are not decorative):

Fill transform — 28ae1fc8. Under TP each rank keeps inter/tp of every expert, so a
paged fill READS the full expert and WRITES a row-gathered subset: read length ≠ write
length. That assumption was baked into fetch_batch_into, which read and copied in one
step — and that, not the gather, is what would have made #527 a painful merge. Now split
into Transport::read_batch + ExpertFillTransform, with None = identity keeping today's
single-step path. A synthetic 1/tp transform pins the properties TP needs (packed_len agrees
with output, ranks partition the expert exactly once, indivisible geometry fails closed), so
only the concrete gather is missing.

Expert ownership — a2c5a940. EP was refused outright, which was over-conservative and
mis-justified. The old guard said "paging must never leave an expert on a zeroed dummy",
true of a paged expert and false of a non-owned one, where the dummy is the mechanism.
The real hazard is the inverse: repointing a non-owned expert makes this rank contribute a
value the owning rank also contributes, and the all-reduce double-counts — wrong answer,
no error. ExpertOwnership makes the rule explicit ("page only what this rank owns") and
plan_dispatch skips non-owned experts in the capacity count, the catalog validation and the
resolve loop.

Also worth recording: paging replaces EP's compaction rather than composing with it. EP
packs owned experts densely via local_of_global; paging assigns slots dynamically. Both
yield base + index·stride, so the pointer-table format is unchanged.

Both guards stay in place. TP and EP are still refused at load, because neither can be
tested on a single GPU — the seams are ready, the activation is not claimed. Each guard
carries its own removal recipe at the rejection site.

Verified: neutrality gate PASSES on the identity/All path (byte-identical to before the
seams), 115 unit tests, guard confirmed to fire.

Caveats I have not resolved:

• TP detection is duck-typed on HIPFIRE_TP / HIPFIRE_TP_EXPERT_SLICE, inferred from
  [Superseded] refactor: unify device mesh, model parallelism, and generation dispatch #527's shape rather than confirmed against its activation path. If it activates
  differently the guard silently won't fire.
• The gather would run per fill under paging, not once at load. On a ~4.7 MB expert
  that's a strided host copy on a path already read-dominated — TP+paging would be correct
  but its cost is unmeasured, and a TP-aware layout at quantise time may be the better
  answer.
• Paging is HFQ-only by construction (the catalog is built from the HFQ tensor index), so
  the safetensors load path has no paging.
• load_weights_inner's expert section will need a hand merge with [Superseded] refactor: unify device mesh, model parallelism, and generation dispatch #527 — architecturally
  compatible, textually overlapping.

[Kaden-Schutt]

Context for anyone picking this up — I started merging this into beta today and stopped, because this PR already landed once and was reverted, and the reason is documented but not linked from here.

What happened on 2026-08-05

c7886d2df  merge: integrate PR #563 DS4 expert paging
0d5e8f99f  Revert "feat(ds4): wire HFP4G32 experts end-to-end — dispatch arms + batched kernels"
52192b56e  Revert "feat(quantize): deepseek4-fp4 — bit-exact FP4 passthrough for ds4 experts"
25111034e  revert: restore pre-PR #563 DS4 staging behavior     (30 files, −7,747)
1289cbbd9  docs(ds4): record post-563-revert golden recertification

25111034e restored the tree byte-identical to the pre-merge parent 6464d6081. expert_pager.rs and alloc_paged_layer_expert_pool do not exist on beta today, which is why this branch now conflicts in 4 files — the merge is not a rebase problem, it is re-applying reverted work.

Why it was reverted

From docs/investigations/2026-08-05-ds4-pr563-revert-golden-recert.md:

PR #563 introduced a real startup concern [...] On the restored binary, identical Redline harness loads were observed both around 90 seconds and beyond the 120-second response timeout.

The recertification confirms the revert restored the golden set: decoded output byte-identical across both serving arms (MD5 e49b9893a207d8a698eb17fdca13db51), DSpark 37.3264 → 37.0646 tok/s (−0.701%), AR retained-PM4 28.8678 → 29.0118 tok/s (+0.499%). Failed 120-second AR load attempts and the post-#563 load failures are preserved in the evidence directory.

That doc is careful to say startup variance is not solely attributable to this merge — "Startup therefore remains a separate measurement and loader-pipeline problem" — so the regression is real but the full attribution is open.

Where that leaves this PR

The decode-path analysis still holds and is good: the new path is gated on expert_quant_type == 21 (HFP4G32), stamped at load from experts.0.w1.weight, and the production SKUs (deepseek-v4-flash-0731.mq2r, .mq2lloyd) are qt 19 enforced by validate_mq2_family_tensor_policy, so they provably never enter the new arm. This was never a decode-correctness problem. It is a load-time problem, which is exactly the class a diff review does not surface.

So what this needs before re-landing is not conflict resolution, it is a load-time measurement against the pre-#563 baseline on the same fixture — specifically that the paged path does not push cold-load past the response timeout. scripts/paging-neutrality-gate.sh was part of the reverted set and would be the natural home for that gate.

Also relevant: since the revert, beta has landed loader work that moves the same ground — c6c3c0ac3 loader: parallelize packed HFQ expert reads, d92ef9144 bench(loader): add bounded parallel HFQ load screen. Those may change the startup calculus that caused the revert, in this PR's favour. Worth measuring against current beta rather than assuming the August result still stands.

Re: @fivetide's question about #527 — @nwoolmer's seam analysis above is still accurate as far as the decode surface goes, but the paged-loading half now has no counterpart on beta at all, so the #527 interaction should be re-checked against that.

Leaving this open and unmerged. cc @nwoolmer

[nwoolmer]

Brought this branch up to date with current master and pushed the result at 57b4bf417.

What changed

• Merged current master, including the slots-era runtime and dispatch changes.
• Restored the PR's paging implementation after accounting for the staging revert already present in master.
• Ported native HFP4 expert dispatch onto the current APIs while preserving:
  • plain-vs-rotated activation handling,
  • current native MQ2 backends,
  • the updated MQ2 down-kernel arguments,
  • resident and paged decode/prefill paths.
• Kept paging explicitly incompatible with heterogeneous routed execution for now, rather than silently placing pools or pointer publication on the wrong GPU.
• Preserved both fp32 and current f16 KV configuration values.
• Wired the expert adapter into registry validation, pull, and removal handling.
• Added explicit cleanup for pager pinned memory/HIP stream/event resources and adapter GPU allocations.
• Updated the KLD helper to use configurable paths and fail closed on empty, incomplete, or non-teacher-forced comparisons.
• Fixed the remaining TP-prefill call site for the new paging-aware ffn_batched signature.

GitHub now reports the PR as mergeable.

Validation

Validation is still pending. Per request, I did not run builds, tests, Hipfire code, examples, benchmarks, or repository gates. The update has only received static diff/API review so far.

[Kaden-Schutt]

Triage: HOLD_WIP. Registry SKUs are already on master; paging/adapter remain unique but were reverted for startup/load timeouts and still depend on the unfinished mesh/TP gather path. Re-land only after the recorded load-time gate clears.

[nwoolmer]

Merged current beta and resolved the conflicts. Head is 2140fbe27. GitHub should now report this mergeable.

This is conflict resolution only. The load-time gate behind the 2026-08-05 revert has not been run, and nothing below claims it has. @Kaden-Schutt's HOLD_WIP still stands on its own terms.

The conflicts were relocation, not disagreement

beta moved 182 commits ahead and refactored three of the four conflicting files out from under this branch. None of it was a semantic clash:

crates/hipfire-runtime/examples/daemon.rs — fe031717d split the 28k-line example into hipfire-daemon + hipfire-generate. Accepted the deletion and ported the three DS4 KLD helpers (ds4_reset_force_tokens, ds4_force_token, ds4_dump_logits) plus their four call sites into hipfire-generate/src/dense.rs, where generate_deepseek4 now lives. Kept the two non-obvious properties intact: the forced-token lookup is keyed on a call counter, not pos (the pre-loop site and the loop's first iteration share a pos, so a pos-keyed lookup hands out forced[0] twice), and the first generated token is forced outside the loop (miss it and the forced model runs one position behind the reference, voiding the comparison).

crates/hipfire-quantize/src/main.rs — beta decomposed the 15k-line monolith into modules while this branch still carried the monolith, so the entire file conflicted as one hunk. Took beta's thin main.rs and redistributed this branch's additions to where they now belong: repack_e2m1_ue8m0_to_hfp4g32 + its bit-exactness test → quant_hfp4.rs; passes_prefix_filter + tests → model_filter.rs; --exclude-prefix → cli.rs; the FP4-passthrough arm, the deepseek4-fp4 / deepseek4-dense-precise format gates and keep_f16_dense → pipeline.rs. Checked first that none of these exist on beta under another name — they don't; they were part of the reverted set.

crates/hipfire-arch-deepseek4/src/forward.rs — beta relocated forward_ep and the mtp_* family into new ep/mtp modules. Took beta's re-exports. The only paging change inside that relocated block was one line — the extra None pager argument at mtp_forward_batched's ffn_batched call — ported into mtp.rs. Kept this branch's five new paging helpers (page_routed_experts, dump_adapter_pairs, prefetch_next_layer_experts, trace_route_lookahead, prefetch_hash_experts) and took beta's widened visibility on ffn_routed/ffn_batched.

crates/hipfire-arch-deepseek4/src/lib.rs — union of both module lists.

Gate hygiene

beta is green on leanup-ratchets and 39/39 on crate maps, so every violation was this branch's, and all three are now fixed:

• Six research probes gated behind --features lab — pinned_smoke, expert_policy_sim, async_prefetch_smoke, fp4_artifact_check, d2h_probe, hfp4_moe_indexed_parity. Confirmed they still compile under --features lab so the gating hides nothing. ungated_examples back to beta's 32.
• Bypass ledger records hipfire-arch-deepseek4 at 21. The +2 are both in this branch's new expert_adapter.rs (gpu.gemv_f16_xf32 for the adapter's A and B projections), declared with a RATCHET-RAISE: trailer — not absorbed silently.
• Regenerated the ten crate maps this branch drifted, including the three new modules and the new half dependency.

leanup-ratchets: OK — 21 metrics, 0 violations. ratchet-diff: no expectation weakened.

Verified

cargo check --workspace --all-targets — 0 errors. Tests pass across hipfire-arch-deepseek4, hipfire-generate, hipfire-runtime and hipfire-quantize (111 + 5 + 10 + 2 + 118).

One failure, pre-existing on beta: reap_overlay::integ::sp4_overlay_to_sp3_load_end_to_end. Worth flagging how it presents — it passes when run alone and fails in the full-suite run, on clean beta as well as here. I initially mis-scored it as a regression by comparing a single-test run on beta against a full-suite run on this branch; the like-for-like full-suite comparison on beta fails identically. It looks like a test-isolation problem in reap_overlay, unrelated to this PR but probably worth its own issue.

What this still does not answer

Everything in @Kaden-Schutt's 2026-08-15 comment stands:

• No load-time measurement. scripts/paging-neutrality-gate.sh was in the reverted set and is not back. That gate — cold-load against the pre-feat(ds4): add DeepSeek-V4-Flash-0731 — routed-expert paging, +28.8% cache budget, and an experimental prefetch adapter #563 baseline on the same fixture — is the thing that has to clear before re-landing, and it needs the GPU box.
• Your point that c6c3c0ac3 and d92ef9144 (parallel/bounded HFQ expert reads) may have moved the startup calculus in this PR's favour is still untested, and is the cheapest next experiment.
• The [Superseded] refactor: unify device mesh, model parallelism, and generation dispatch #527 interaction should be re-checked: my earlier seam analysis holds for the decode surface, but the paged-loading half has no counterpart on beta.

Leaving as draft.