Conversation
Kaden-Schutt
left a comment
Kaden-Schutt
left a comment
There was a problem hiding this comment.
Maintainer review — merged onto current beta (65a4d826a) and measured
Note first: g4/integration is now identical to beta. The G4 line landed (#747 included) and #730 is review-ready, so this PR effectively targets beta. The merge is clean apart from two additive conflicts (admission.rs: your qwen35 EP-MoE refusal + my pp/tp geometry gate; qwen35_reset_hw.rs: your extra session_generate arg + my expect_dflash). Resolved by keeping both; builds clean.
What passes on the merge
qwen35_moe_single_route_oracle— the single-device sealed route is bit-exact against itself from fresh loads on the ornith pin (64.9 s).ratchet-diffOK, env-docs OK.check-dispatch-bypass: you paid down one bypass inhipfire-arch-qwen35(137 → 136). The only ratchet failure isbypass_slack = 1, i.e. the ledger row indocs/governance/debt-dispatch-bypass.txtneeds lowering to bank it. Please do — that is a win, not a defect.- Crate map:
hipfire-runtime/src/sealed_moe.rshas no map entry;python3 scripts/check-crate-maps.py hipfire-runtimeregenerates it.
The scope question, stated plainly
This PR seals the single-device MoE route and hard-refuses EP: shard_moe_experts / shard_all_moe_layers / the compact EP launch all return "sealed MoE EP execution is unsupported" (forward.rs:3145–3177). Both G5 route oracles on beta therefore fail at load rather than at comparison, and even ep_decode_parity tp=1 refuses at layer 0.
That is consistent with your own summary ("does not claim G5 acceptance, physical multi-GPU/RCCL proof"), and single-device sealing first is a defensible order. But it means the two EP oracles on beta stay red for a different reason than before — no route rather than divergent route — and the G5 acceptance bar is unchanged: qwen35_ep2_vs_single_oracle and qwen35_ep4_vs_ep2_oracle green at assert_eq-class.
We measured, on beta before this PR, exactly what sealed EP has to solve: sharded EP diverges from single by a top-k flip seeded by the association difference (EP sums per-rank expert-parity groups; single folds the k=8 slots in slot order). Canonical rooted reduce, EF residuals on the multi path, and f32 recurrent state each failed to close it — only a slot-order combine can. Full plan with file:line, unit breakdown, and gates: /home/kaden/hipfire-g4-val/G5-SLOT-COMBINE-PLAN.md (I can post it here if useful). If your sealed executor's combine semantics take that shape, the EP half becomes a small follow-on rather than a redesign.
Fixture
All live evidence here is on qwen3.6:35b-a3b / Qwen3.6-27B on gfx1151. Those artifacts were retired on 2026-09-11; the pins are ornith-1.5:35b-a3b-mq4r (MoE, sha256 84103fcc…) and qwen3.8:27b-mq4-xt (dense, 9f91556f…), and the acceptance host is the 4× R9700. I have run the single-route oracle on the pin above; the matrix rows and PM4 parity on the pin are what I'd run next once the two gate items land.
Requested before merge
- Lower the bypass ledger row (bank the −1).
- Regenerate the
hipfire-runtimecrate map. - Rebase onto
beta(or I can push the resolved merge to your branch — say which).
Then I run the full matrix + Redline on the ornith pin on gfx1201 and merge to beta as a G5 constituent (single-device sealing), with EP explicitly still open on the tracker.
Keep PR755 source-bound expert ownership while lowering sealed calls into fixed-capacity computation stages and sharing the named collective schedule across Qwen root-routed decode and batched prefill. Preserve kernel selection, reduction order, route proofs, and existing diagnostic policies. gfx1151: 10258 chunk-1 observations byte-identical to d9b05c4; chunk-16 retains the same baseline warm-prefix mismatch with identical raw arrays. Stable retained capture: 862 launches, 22 kernels, fingerprint 229e1cdc8cdbb0aa; 15-position PM4/HIP and GDN-frame parity exact. Five-turn battery and chain transcripts match baseline; three fresh-process expert-mutation rollback/recovery rounds pass. Affected library tests and seven new schedule tests pass. Default-parallel workspace run hit the unchanged rdna-compute exact-VRAM assertion; serialized GPU-locked rerun and remaining suites pass. Narrow clippy completes with warnings. Repair two stale AR fault-test helper calls. No physical EP2/EP4, full G5, performance, or product PM4 admission claim. See docs/design/sealed-granular-moe.md.
…, slot tool turns (#753)
v0.3.1: promote beta — DFlash repairs, Gemma/Maple, image and vision support
Centralize Qwen and Cohere MoE sequencing in sealed dispatch programs and keep architecture crates focused on typed resource binding. Simplify root-routed EP execution while preserving collective ordering and receipt validation.
fivetide
changed the title
|
Pushed This supersedes the prior single-device-only architecture: Qwen decode/prefill/compact EP now enter the common root-routed schedule, and Cohere consumes the same sealed granular Step machinery through its sigmoid/no-shared recipe. Architecture-local MoE stage interpreters were removed; crate maps were regenerated. Local gfx1151 proof is complete for build/tests, Qwen and Cohere MQ4 numerical/state parity, serving transcript retention, Redline shadow parity, and injected rollback/recovery. Physical EP2/EP4 remains explicitly blocked on qualifying distinct gfx1201 devices; no emulated proof or G5 admission claim is being substituted. Cohere MQ6/Q8 candidate qualification is separately blocked by current host/UMA OOM, as documented in the PR body. |
fivetide
added
the
ratchet-raise
Merge current upstream master, refresh generated maps and dispatch debt, and repair newly denied clippy findings. RATCHET-RAISE: daemon_lines 4840 -> 4863, traded for fail-closed max_seq admission and merged daemon lifecycle fixes.
|
CI repair pushed as 1db83e0.\n\nChanges:\n- merged current upstream master (2774aa2) and regenerated the three conflicted crate maps\n- fixed the denied approximate-constant lint in the hidden-scatter example and the constant-index clippy errors in diffusion\n- banked Qwen35 dispatch-bypass reduction (136 -> 131), removed the stale zero-debt Cohere2MoE row, and lowered total debt 257 -> 245\n- refreshed all crate maps affected by the merged tree\n- added the required ratchet-raise label; the existing daemon ceiling trade is declared in the merge commit\n\nLocal verification:\n- cargo clippy --workspace --all-targets --locked: pass (warnings only)\n- cargo build --release --workspace --all-targets --locked: pass\n- HIP_VISIBLE_DEVICES=-1 ROCR_VISIBLE_DEVICES=-1 cargo test --lib --workspace --locked: 2,949 passed, 38 ignored\n- bash scripts/leanup-ratchets.sh: 21 asserted metrics, 0 violations\n- python3 scripts/check-crate-maps.py --check: 44 maps match\n- CI-equivalent ratchet-diff against upstream/master: pass |
Add diagnostic HIPFIRE_EMULATE_GPUS at the existing device resolver and sealed admission seams; keep product topology gates unchanged. Canonicalize rank-local grouped slots before root gathering and overwrite reused shared-down scratch instead of accumulating stale rows. On one gfx1151: EP2/single and EP4/EP2 full logits byte-identical at all 27 positions; EP4 batch lifecycle passes. TP2 tokens match at 27 positions (max logit delta 0.2968, existing bound 1.0). Kernel channel: 17 passed. Cargo build and workspace tests pass with the known unrelated glimmer_q8_classes_narrowing_keeps_lm_head_and_embed_only test excluded; scoped clippy completes with existing warnings. Formatting, env docs and four crate maps pass. Logical-rank evidence only: no physical EP, RCCL, interconnect, performance, product gfx1201 or G5 admission claim.
Required hardware validation before physical EP / G5 acceptanceThis PR is ready for review, but the new Halo results exercise logical ranks on one physical gfx1151 only. They do not replace multi-device validation. Remaining hardware work:
RCCL remains unvalidated and outside this emulation port's verification scope. Physical peer-path passes must not be presented as RCCL proof. No throughput, product gfx1201 admission, product PM4 admission, or G5 acceptance is established by the logical-emulation results; those claims still require their claim-scoped validation/admission evidence. |
2 participants
Summary
Stepprograms owned byhipfire-dispatchhipfire-runtimefor decode, prefill, and batch paths while preserving transport precedence and using root-gathered global slots followed by the ordinary canonical combine for Qwen EPThis implements the approved shared-step G5 direction on top of
cdbe19bd23bd5b34b19e3e285a10b37e3e6f5670. It intentionally does not introduce a DSL, generic DAG scheduler, owning cross-family model wrapper, or new product admission.Latest implementation commit:
27abd53e4.Architectural result
Qwen and Cohere now declare one of two typed recipes:
SoftmaxGatedSharedSigmoidRoutedNoSharedThe shared lowerer creates opaque
SealedMoeOpoperands and executes the resulting borrowedStepprogram through the existing validated launch loop. Route readiness remains invocation-local; device/context identity and receipt identity are checked before consumption. Existing model, scratch, KV, recurrent-state, and generation ownership remain unchanged.The root-routed collective runner now performs its fixed sequence procedurally instead of interpreting a second command list. Lease metadata and scratch capacity are validated before mutation and again at reduction.
Earlier shared-step verification
Passed on local gfx1151:
cargo build --release --lockedcargo test --workspace --lib --locked -- --test-threads=1cargo check --locked -p hipfire-dispatch --no-default-featuresgit diff --checkBehavioral comparison against freshly preserved baseline binaries:
Explicit blockers / non-claims
hipMallocOOM at layer 29 expert 68 under current host/UMA pressure. MQ4 covers scalar, grouped, and fresh/scalar Q8 router policy behavior exactly, but this does not claim MQ6/Q8 live qualification.Review notes
gate_up_krather than the flat backing allocation dimension, eliminating the observed GPU fault without changing kernel semantics.Logical GPU emulation and canonical EP validation —
27abd53e4Diagnostic
HIPFIRE_EMULATE_GPUSis now supported at the existing runtime device resolver and sealed ownership checks. Values >=2 enable modulo physical-device aliasing; constructors still choose logical rank count. Same-device ranks retain separate streams, buffers, and ownership, and report peer access disabled. The batch exception admits only the exact four-logical-rank/single-gfx1151 diagnostic shape; product topology admission is unchanged.Strict oracles exposed and now cover two arithmetic/state bugs:
[token * k_top + slot, dim]order before gather; root uses the ordinary canonical combine.Results on Halo (one physical gfx1151)
EP comparisons also require finite logits and within-mesh route/post-combine residual byte agreement. No EP tolerance was introduced. Temporary investigation diagnostics were removed; permanent assertions remain.
Fixture identities verified by the oracles:
ornith-1.5-35b-a3b.mq4r, SHA-25684103fcc8ade42aa2ac8ec01176df7a4ead5e94810597c9fae2f6763152a3ac6.qwen3.8-27b.mq4-xt, SHA-2569f91556f7e0431a077d03756a7102d0154108757289e6e5fe9a2d204c0c9eeb7. The current upstream file had been reissued; the exact pinned bytes were recovered from immutable HF revision8f573974425d2ffbc9754be87bab2e90fba0e523. No fixture pin was relaxed.Reproduction
All GPU runs were serialized with
flock -w 3600 /tmp/hipfire-gpu.lock. The cleaned EP matrix was run with:That invocation passed all three EP oracles; TP2 initially failed because its pinned fixture was absent. After recovering the exact fixture, TP2 passed separately with
HIPFIRE_EMULATE_GPUS=2and the exact filterqwen35_tp2_vs_single_oracle -- --ignored --exact --test-threads=1 --nocapture. The emulation value is only an enable switch, so the matrix still constructs the requested EP2 and EP4 rank counts.Additional verification for this commit:
cargo build: PASS.cargo test --workspace -- --skip diagnostics::tests::glimmer_q8_classes_narrowing_keeps_lm_head_and_embed_only: PASS. The previously known unrelated quantize failure is explicitly excluded, not fixed or claimed passing.scripts/check-env-docs.py, and crate-map checks for runtime, Qwen35, dispatch and rdna-compute: PASS.Evidence boundary: logical EP emulation on one gfx1151 validates ownership, route broadcast, canonical contribution gathering/combine, result broadcast and lifecycle sequencing. It does not validate RCCL, PCIe/xGMI, physical-device synchronization, physical EP2/EP4, throughput, product gfx1201 admission, or G5 acceptance. Physical multi-GPU validation remains blocked.