G4.3–G4.6 lifecycle and load ownership stack

[fivetide]

Summary

This is the upstream-native grouped G4.3–G4.6 implementation/evidence stack, based on merged beta at 5773f62497d603ef8678ac09beb72869517a0ee2; the author's final head was 31682603f6265b81f8c3674a7024084f751ef1d2 (evidence-only amendment atop tested source head 17f84e57ecf2f4381aabc414670eefe7657718ce). It is intentionally a constituent PR for tracking issue #666, not a milestone-promotion claim. The current head is 646f5a30c — the original stack plus a merge of beta 6208dec39 and the maintainer repair wave described below. Singleton terminal ownership remains fail-closed when inactive or mismatched. One wire-contract change: a generate envelope with attempt_id: 0 was accepted by the beta daemon and is now rejected with a typed validation error (hipfire-daemon/src/main.rs, require_wire_attempt_id). Zero is reserved for uncorrelated control-plane envelopes so a terminal tagged 0 can never be confused with a live attempt. The first-party client already numbers attempts from 1 and surfaces the rejection; only direct stdio-protocol users hard-coding zero are affected. Missing or malformed attempt_id was already invalid.

Tracking: Part of warpfront/hipfire#666; no Fixes/Closes keyword is used. Merge state, constituent evidence, and G4 milestone acceptance remain separate.

Exact G4.3–G4.6 scope

G4.3 — Direct Qwen35 model-load transaction

• Eliminate the untransactional model_load::load_weights route still consumed by direct qwen35::load_weights.
• Migrate callers to the existing transactional owner contract rather than adding a second transaction design.
• Required final-head evidence: fault after embedding, completed layer, final norm, and output-publication boundaries; exact rollback/retry accounting; production Qwen35 load/generate/reset/unload/reload with the retained carrier/store route unchanged.

This stack carries the direct Qwen35 transactional load cutover plus staged ownership/rollback/retry coverage. Final composed-beta reruns remain required for milestone acceptance.

G4.4 — DeepSeek4 DSpark load transaction

• Stage load_dspark layer and global owners through ordinary late errors using current DeepseekV4LayerWeights::free_gpu.
• Keep the existing injected DsparkHead wrapper as a fault seam, not as the complete transaction.
• Required final-head evidence: real late layer/global/helper failure, immediate retry, AR/DSpark output preservation, reset, unload/reload.

The four owner-fault seams remain the explicit G4.4 manual fault inventory:

1. dspark_after_layer_fault_rolls_back_and_retries
2. dspark_after_head_helper_fault_rolls_back_and_retries
3. dspark_after_main_proj_fault_rolls_back_and_retries
4. dspark_after_global_fault_rolls_back_and_retries

They were executed on the final tested code head (17f84e57ecf2f4381aabc414670eefe7657718ce) on the available gfx1151 GPU. The canonical full 82GB DS4 serving battery remains blocked; it was not rerun.

G4.5 — Terminal-safe parser finalization

• Add terminal latches to both EosFilterState and ThinkOutputRouter.
• Make finalization idempotent; late observe/push is inert; reset restores reuse.
• Preserve flush_pending as a compatibility alias so AR, Qwen35 spec, and VL callers do not churn.
• Retain consuming ToolOutputRouter, QwenArSemanticProducer, and DeepSeek DSML finalization outside this port.

Permanent state-machine and malformed/open-think/length-cap semantic coverage is included. The contract is no normal done, tool release, cache publication, or late bytes after terminal finalization.

G4.6 — Exactly-once terminal ownership

• Carry the whole terminal-cardinality boundary: engine (request_id, attempt_id) terminal claims and gates for staged done, correlated error, and cancellation.
• Carry the GenerationRoute::ALL lifecycle adapter/start latch and route terminal wrappers.
• Migrate direct abort/error bypasses in Qwen EP, DS4 heterogeneous, Glimmer, and dense error emission.
• Add keyed wire-terminal claims for continuous-batch AttemptKey entries; singleton claim state does not cover batch lanes.
• Preserve exact commit_ready correlation and Commit-gated assistant-cache/tool effects; do not redesign them.

Required final-head evidence is concurrent done/error/cancel races; duplicate/late/wrong-attempt writers; one start and one terminal per route; independent batch-lane claims; valid attempt-zero pre-admission errors; and normal commit, abort, timeout, cancellation, and next-request reuse on the production daemon protocol.

Commit stack

In base-to-head order:

• 9557ed454be07e0342294ebd6fac6a5ed79dd5ed — fix(runtime): make parser finalization terminal-safe
• 3951906cbe2495864cb4394c1d14e639ef16f1e1 — fix(lifecycle): claim terminal ownership across generation routes
• 8433558bd8daec157802de43eba26db7a68044e5 — fix(runtime): reset think router for fresh turns
• 382290add90a1aa919827113298f21bd51ae8704 — fix(lifecycle): key terminal generations and route cancellation
• fe43cf4c83d657e2eede6f47e44995333d44509b — fix(lifecycle): fail closed inactive singleton claims
• 08361528c891801b9aa574f98a46564e4010e129 — fix: transact direct Qwen model loads
• d3f891f4722b8024e44f04c3c3f526abe3db0d81 — fix: transact DeepSeek DSpark sidecar loads
• a93731380c2d44c90b7887156caaf7605cb7c856 — fix: stage llama and paro layer owners
• d90d33d78ba5b4f1f020887f3f14eb1103ff636d — test: cover staged rollback reuse and dspark retries
• 1be56c398d464d785bc4a812875eb817b23997eb — test(lifecycle): activate singleton terminal owners
• 1b4f70d2a6c6bda0865b973ac80868abb4a92296 — fix(batch): retain terminal registry through done emission
• a990c363a998f4336a51a613d1229998d39be038 — fix(route): map Qwen dense TP EP to semantic AR
• 5342bf9c65d18b6107a0f84e806ea33411fbe063 — fix(vision): scope standalone generation routes
• 4b69a8aa4b2d5c2ea7811df223426aae8d30332a — fix(daemon): correlate admission errors
• f28792aefacd24df35cc362913d752ca739508df — fix(protocol): reject zero generation attempts
• ecb618a486e2bab2fd898164406b91bfdd12beca — fix(batch): guard cleanup by admission generation
• 77e0165dc89ea22b87717eb5f1958d6e59606dd4 — fix(daemon): announce startup generate before admission
• 8c03d24db79c49ba685046872612a2e38695dae5 — fix(batch): reject reserved zero attempts in drivers
• 0f37db0be67bc13c2a75e2b68368ed99822e5d36 — fix(batch): correlate direct inbox admission errors
• 1c6657980a23f3f86936c91137a8f31f741d3f88 — fix(route): own standalone vision generation starts
• 40321d85ff89f2dc5834d995922ec5549fefb6bc — fix(batch): carry admission token into drivers
• 009226109e927eee9f1ae415fbb6fd7d3636e2fb — fix(batch): correlate lfm assignment capacity errors
• bdc55ade282134ec585f1ad64593dcba3933f739 — style(batch): format blocker regression
• e02de897b9c2cae253a4325fe047fa575a6fbc36 — fix(batch): release lfm route latch on assignment error
• 1e40c85ef0986c1d71d4e0a2b236be89a53f617b — fix(batch): preserve singleton ownership across think barriers
• 65063a19acc90d30dadaa3dfba715c3a1dae7d5e — fix(batch): isolate singleton handoff scope from reused keys
• 9de160eab62aaf998717754719ba745e9c3066ab — fix(batch): use explicit routes for terminal latches
• c36fa73cbed3e8ee02d4ced5cb0d16613fd37e60 — fix(generate): route post-start terminals
• 7e64341b1069e1395806b8f2d6d99b01fa01fcf0 — fix(batch): bootstrap singleton handoff owner
• 2d75e75f197b72882fa6a91fcdc9d5e599c96080 — fix(terminal): serialize singleton handoff ownership
• 08a59dcbdc8feeb00d37c67a7a7a9ae34725c183 — fix(slots): clean up qwen route terminals
• 42580d6091e8308268409735e20c28d47126ec7c — fix(route): own pipeline generation lifecycle
• 6c57ee7a822f8d1c314a5a22818d92b91fda2e2b — fix(batch): flush route terminal envelopes
• 17f84e57ecf2f4381aabc414670eefe7657718ce — fix(batch): release latches after terminal claim (final head)

Adversarial blocker remediation

The complete 1be56c398..17f84e57e stack closes the reviewed blockers while preserving constituent/nonpromotion framing and direct-emitter visibility semantics:

• Batch done ordering: 1b4f70d2a retains the exact terminal registry through done emission so claim, write, and cleanup ordering cannot retire a live batch owner early.
• Dense TP route mapping: a990c363a maps Qwen dense TP/EP into the semantic AR route instead of allowing a terminal path to bypass route ownership.
• Vision scopes/start: 5342bf9c6 and 1c6657980 scope standalone vision generation and own its route start exactly once.
• Post-admission correlation and zero rejection: 4b69a8aa4, f28792aef, 77e0165dc, and 8c03d24db correlate admission failures, announce startup generation before admission work, and reject reserved zero attempts.
• Generation-token propagation and cleanup: ecb618a48, 40321d85f, and 0f37db0be carry the opaque admission generation into drivers and guard cleanup/direct inbox errors against stale owners.
• Direct batch errors: 0f37db0be ensures direct inbox/admission errors use the exact keyed batch terminal rather than an uncorrelated singleton path.
• LFM assignment error and latch: 009226109, bdc55ade2, and e02de897b correlate assignment-capacity errors, retain the regression formatting, and release the LFM route latch on that terminal.
• Think-barrier payload/owner/tombstone/abort/same-key races: 1e40c85ef, 65063a19a, 7e64341b1, and 2d75e75f1 preserve full handoff payloads, isolate owner scopes, retain tombstones, serialize abort/commit ownership, and prevent reused-key overtakes.
• Per-key batch route terminals: 9de160eab makes route/latch selection explicit for each (id, attempt_id) batch lifecycle.
• All post-start route bypasses: c36fa73cb, 08a59dcbd, and 42580d609 route post-start terminals through the production adapter, including slots and PipelineParallel.
• Terminal flush visibility: 6c57ee7a8 flushes route terminal bytes after staging so successful batch done/error/cancel envelopes become visible.
• Claim versus delivery on failed output: 17f84e57e separates claim consumption from visible delivery; write/flush failures release the exact start latch, while duplicate or unclaimed writers remain suppressed and direct bool APIs still report delivery.

The permanent regressions cover active-route clearing, claim consumption, duplicate suppression, fresh same-key starts, and successful done/error/cancel behavior. This section is remediation evidence only; it does not promote G4.

Historical pre-merge validation evidence at 17f84e57ecf2f4381aabc414670eefe7657718ce

This section records pre-merge validation of the tested source head and is historical context, not a current branch-head acceptance claim. Current final-head evidence is linked in the closeout below.

Actual gfx1151 dual-lane smoke

This was a real isolated daemon run on Strix Halo gfx1151 using /home/bjoern/.hipfire/models/qwen3.5-4b.mq4, VMM KV, continuous batch size 2, MTP/DFlash off, and rebuilt binaries from this branch. The raw JSONL recorded:

• One gen_start for cb-fixed-c, attempt 501, and one for cb-fixed-d, attempt 502.

• Tokens for attempt 501: Hello, there, ! (3 tokens). Tokens for attempt 502: Until, we, meet (3 tokens).

• commit_ready for 501: tokens=3, continuous_batch.executed=true, lane=0, slots=2, max_active_lanes=2.

• commit_ready for 502: tokens=3, continuous_batch.executed=true, lane=1, slots=2, max_active_lanes=2.

• The client sent these exact commits:

  {"type":"commit","id":"cb-fixed-c","attempt_id":501}
  {"type":"commit","id":"cb-fixed-d","attempt_id":502}

• Both exact done envelopes followed, retaining attempts 501/502, lanes 0/1, slots=2, and max_active_lanes=2.

• The final protocol event was {"type":"unloaded"}. The daemon stderr recorded both exact commit receipts.

Raw evidence captured during validation: /tmp/g4-final-qwen-raw-fixed4.jsonl and /tmp/g4-final-qwen-stderr-fixed4.log.

No-model and attempt-zero validation

The final daemon protocol emitted these exact lines before a successful unload:

{"type":"error","message":"no model loaded","class":"validation","retryable":false,"rolled_back":false,"attempt_id":123,"id":"no-model-fixed"}
{"type":"error","message":"generate attempt_id must be nonzero","class":"validation","retryable":false,"rolled_back":false,"attempt_id":0,"id":"zero-fixed"}
{"type":"unloaded"}

Raw evidence: /tmp/g4-final-no-model-fixed.jsonl.

Vision limitation probe

A Qwen35 text-only load reported vl:false. An image request using a nonexistent path was rejected before image access with the exact result:

{"type":"error","message":"model has no vision encoder","class":"internal","retryable":false,"rolled_back":false,"attempt_id":601,"id":"vision-fixed"}

Unload succeeded. This validates the non-vision limitation only; it is not actual VL-model success evidence. Raw evidence: /tmp/g4-final-vision-fixed.jsonl.

Automated results

• cargo test -p hipfire-generate --all-targets: 255 passed across 22 suites.
• cargo test -p hipfire-engine --all-targets: 91 passed across 5 suites.
• Route lifecycle filter: 22 passed; the focused route flush regression and engine lane-reuse regression also pass.
• cargo build -p hipfire-engine -p hipfire-generate -p hipfire-daemon -p hipfire-client --all-targets: passed with existing warnings.
• cargo clippy -p hipfire-engine -p hipfire-generate -p hipfire-daemon -p hipfire-client --all-targets: passed with existing warnings.
• git diff --check: passed before publication; branch is clean at the published head.

Formatting baseline

The scoped rustfmt check was run on the changed engine/generate files. It still exits nonzero because the repository has pre-existing formatting differences in terminal.rs and older batch.rs lines (module-recursive rustfmt reports those baseline diffs). No rustfmt diagnostics remained for emit.rs, ar.rs, or the newly added batch failure regressions, and no broad formatter churn was retained.

The protected release process on 127.0.0.1:11524 and its daemon were left untouched throughout validation.

Maintainer repair wave (pushed to this branch, 31682603f → 646f5a30c)

Addresses the three code blockers from the maintainer review; see the review comment for citations.

• bf764df17 — gen_start is flushed on the real writer (define_route_start and the deepseek4 EP adapter previously flushed only a Vec); new emit_aborted_terminal_after_abort helper in hipfire-engine, and all seven ClientTerminalDecision::Abort arms (LLaMA, MiniMax, Cohere, Qwen2, Maple, Dots OCR ×2) now emit one attempt-correlated aborted terminal instead of nothing, so the client drain unblocks. Route-count assertion replaced with a flush-observing per-route test.
• e632b312d — DSpark rollback tests now exercise the production seam DeepseekV4::load_dspark_with_fault on the real deepseek-v4-flash-0731-dspark.mq2r (HIPFIRE_DSPARK_FIXTURE) at AfterLayer(0)/AfterHeadHelper/AfterMainProj/AfterGlobal: assert Err, free VRAM back to baseline, retry clean, head GEMV to finite logits. The synthetic zeros-owner tests are gone.
• b32b2321d — gfx1201/ROCm 7.15 permanently reserves 153,092,096 B driver-side on the first hipMalloc of a process (malloc/free ledger balances to zero after rollback; a second fault load adds 0). Tests warm that reservation by name before baselining; no slack change.
• 82f852a9c, e5df0065c, 646f5a30c — RATCHET-RAISE: daemon_lines 4411 -> 4816 (admission correlation, zero-attempt reject, startup announce live in the daemon loop) and crate-map refresh.

Hardware receipts, repair head vs beta 6208dec39

gate	host	result
Oracle: Qwen3.8-27B, ornith A3B, LFM-1.2b — AR battery + chain, greedy	R9700 gfx1201	30/30 turns byte-identical content and cached_tokens
gen_start delivery	R9700	TTFB ≈0.3 ms vs 5.65 s prefill, both arms
Lifecycle: mid-stream cancel → follow-up; restart	R9700	exactly one abort terminal for the cancelled id; follow-up stop; clean after restart (no HTTP reset/unload route exists)
DSpark seam tests on the real sidecar	R9700	4/4
Qwen3.8 AR + DFlash battery/chain	RX 7900 XTX gfx1100	5/5 all; chain cached 0/284/493/674/939
ornith A3B q8 battery + chain	XTX	5/5, 5/5
DS4 deepseek-v4-flash-0731.mq2r + DSpark battery	Strix Halo gfx1151 (103 GB)	loads, sidecar paired, 5/5 stop on both arms; DSpark τ<1 and empty code/math turns are identical on beta (drafter quality, tracked separately)

Observed but not attributable to this PR: lfm2.5-8b-a1b.mq4 on gfx1100 q8 greedy hits attractor 3/5 on beta and on this head alike.

Still open from the ledger below and deliberately not claimed here: direct-Qwen embedding/layer/norm/output fault boundaries, a malformed-tool-call fixture, a successful VL route. They remain G4.3/G4.4 follow-ups; the oracle above is what establishes that the ordinary serve path is unchanged for existing users.

Residuals and explicit nonclaims

• This is a constituent PR only; it does not promote G4.3, G4.4, G4.5, G4.6, or aggregate G4. Final acceptance still requires the complete composed current-beta matrix, exact receipts, and one milestone-level revert identity under #666.
• No physical multi-GPU or distinct-GPU proof is claimed; the gfx1151 smoke is a single physical GPU with two independent batch lanes.
• No full 82GB DeepSeek4 serving battery is claimed; the four manual DS4 owner-fault seams were executed on final tested code head 17f84e57ecf2f4381aabc414670eefe7657718ce, but canonical full-route acceptance remains blocked.
• No actual VL-model success is claimed; the vision probe loaded a non-VL model and only demonstrated the expected rejection.
• The LFM batch positive route remains an unreachable/coverage concern; no LFM batch success claim is made.
• Id-only singleton abort remains normal serialized behavior; it is not evidence of batch-lane parallelism or a distinct lifecycle proof.
• Historical battery/chain counts from earlier stack validation are not reused as final-head acceptance evidence here.

Final-head evidence closeout (constituent only)

The amended evidence is committed at 31682603f6265b81f8c3674a7024084f751ef1d2: README.md and ledger.jsonl. Closed constituent routes are Qwen3.5 lifecycle/G4.5 semantics, singleton and corrected continuous-batch G4.6 lifecycle/cancellation, semantic DFlash battery/chain, and the ten module-qualified GPU fault seams (four runtime DFlash, four DSpark, and two Qwen35 owner-reclaim), with abort-contract postchecks recorded.

Disposition remains constituent-only; aggregate G4 is not promoted. Explicit blockers and nonclaims are: one physical GPU with no RCCL multi-GPU proof; local DeepSeek4 target/draft digest(s) do not match the canonical full-82GB fixture and no canonical full route ran; absent canonical Qwen3.5 A3B; production direct-Qwen fault injection does not cover embedding/completed-layer/final-norm/output-publication boundaries; no deterministic canonical malformed-tool fixture; no canonical accepted VL fixture (available explorations failed); no numerical/mutable-state oracle and no aggregate G4 promotion; no performance or quality promotion; protected-listener before/unchanged/no-signal remains an operator observation rather than independently committed evidence; and strict clippy/rustfmt baseline issues remain. The initial continuous-batch failure remains an environmental cache-packaging observation, not a route rejection.

[fivetide]

Adversarial review closure — final head 17f84e57ecf2f4381aabc414670eefe7657718ce.

The HIGH route-latch blocker is remediated: terminal outcomes now separate claim consumption from visible delivery, so batch write/flush failures release the exact start latch while duplicate/unclaimed writers remain suppressed. Added write-error and flush-error same-key reuse regressions; successful done/error/cancel behavior remains covered. Final affected tests: generate 255 passed, engine 91 passed; build and clippy passed with baseline warnings. This remains a constituent G4.3–G4.6 PR only; it does not promote G4.

[fivetide]

Correction to the publication record: the PR body now enumerates every commit from 1be56c398..17f84e57e through final head 17f84e57ecf2f4381aabc414670eefe7657718ce (24 exact entries, including 1b4f70d2a, 2d75e75f1, and 42580d609). The grouped adversarial-remediation section now covers batch done ordering, dense-TP mapping, vision start scopes, admission/zero rejection, generation-token cleanup, direct batch errors, LFM assignment/latch, think-barrier ownership/tombstone/same-key races, per-key terminals, slots/PipelineParallel bypasses, flush visibility, and claim-vs-delivery latch release. Evidence and nonclaims are preserved.

[fivetide]

Constituent-only final-head closeout: amended evidence commit 31682603f6265b81f8c3674a7024084f751ef1d2 records Qwen3.5 lifecycle/G4.5/G4.6, corrected batch/cancellation, DFlash semantic routes, and all ten final-head GPU fault seams. Aggregate G4 is not promoted; the README/ledger list the one-GPU/no-RCCL, DS4 full-82GB/digest, missing A3B, incomplete direct-Qwen boundary, malformed-tool, VL, and oracle blockers plus all nonclaims.

[Kaden-Schutt]

Maintainer review of 31682603f against beta 07d08aa72 (merge-tree is clean; the blockers below are in the head itself, not merge fallout). Two of them I verified directly in the diff.

Blocking

1. gen_start lost its flush. define_route_start (crates/hipfire-generate/src/ar.rs:1129-1140) renders via emit_gen_start into a Vec — so the flush inside emit_gen_start (crates/hipfire-engine/src/emit.rs:69-82) flushes the Vec — then write_alls to the real writer and never flushes it. On piped daemon stdout that can hold gen_start back through the whole prefill, violating the ordering documented at qwen.rs:1048-1059. Every migrated route is affected. Needs a real-output flush plus a test whose writer observes flush, not a Vec cardinality check.

2. G4.6 exactly-once terminal is not complete. await_client_terminal_commit returns Abort without emitting any terminal (crates/hipfire-engine/src/terminal.rs:1277-1318), and the LLaMA, MiniMax, Cohere, Qwen2, Maple and both Dots OCR loops discard that decision (ar.rs:5203-5209, dense.rs:7357-7362, :8101-8106, :8334-8339, :9038-9043, vision.rs:2281-2286, :2476-2481). The first-party client then blocks draining until a correlated aborted-done/error (hipfire-client/src/lib.rs:1009-1065). The GenerationRoute::ALL adapter tests do not establish one terminal per production route.

3. G4.4 evidence does not exercise the production fault seam. The seam is DeepseekV4::load_dspark_with_fault (crates/hipfire-arch-deepseek4/src/arch.rs:2803-2816); the four named tests build one-element gpu.zeros owners and call DsparkLoadStaging::rollback / pool reuse (arch.rs:3887-3949) — no fixture, no seam — yet the evidence README records them as DSpark rollback/retry passes (docs/investigations/evidence/2026-09-10-pr742-final-head/README.md:133-135). Either drive them through the production loader with immediate retry, or don't present G4.4 as complete.

4. The ledger itself lists open G4.3/G4.4 obligations (README.md:173-185): canonical DS4, A3B, direct-Qwen embed/layer/norm/output fault boundaries, malformed tools, VL, and a numerical/mutable-state oracle. The CPU transaction sweep (model_load.rs:570-668) validates the generic owner algorithm, not the direct-Qwen GPU boundaries. These are constituent obligations, not aggregate-promotion extras.

Non-blocking

• This is a wire cutover: attempt_id: 0 was accepted by the beta parser and is now rejected (hipfire-daemon/src/main.rs:165-175, 2261-2275). The first-party client handles it (hipfire-client/src/lib.rs:769-815); older/direct clients using zero stop generating. Say so in the body rather than "no compatibility weakening".
• generation_route_matrix_tests.rs:858-863 pins route cardinality, not behavior. Replace with end-to-end start/flush/terminate coverage per route.
• No beta-vs-head A/B that ordinary single-slot Qwen token bytes, cached_tokens, and commit payloads are unchanged; start-delivery timing did change (item 1).

What we will run here once the above is addressed

gfx1100 and gfx1201: load battery + serve battery/chain (AR and DFlash) on the pinned Qwen dense / A3B / LFM fixtures; beta-vs-head bytes and cached_tokens; commit/abort/timeout, reset, unload/reload; production load-fault/retry. gfx1201 additionally: 4× EP continuous-batch/RCCL races (daemon/main.rs:3082-3085), DS4 dense TP3/TP4, canonical DSpark load/fault/retry, Qwen VL and Dots OCR. The gfx1151 campaign is welcome context but does not cover those.

Beta promotion (#730) waits on this PR by the maintainer's decision, so a fix wave here is the critical path. Happy to re-review on push.

[Kaden-Schutt]

Maintainer repair in progress on repair/742-lifecycle (PR head + current beta 6208dec39, clean merge):

• bf764df17 — blockers 1 and 2: define_route_start and the deepseek4 EP start adapter now flush the real writer; new emit_aborted_terminal_after_abort helper in hipfire-engine, all seven Abort arms route through it (attempt-correlated aborted + done(finish_reason=aborted), exactly-once via the wire-terminal claim). Route-count assertion replaced with a flush-observing per-route test; engine + generate suites green.
• e632b312d — blocker 3: the four DSpark tests now go through load_dspark_with_fault on a real sidecar (HIPFIRE_DSPARK_FIXTURE), fault at AfterLayer(0)/AfterHeadHelper/AfterMainProj/AfterGlobal, assert Err + free-VRAM recovery, retry clean, head GEMV to finite logits. Ignored by default.

Next: run those four on gfx1201, then the gfx1100/gfx1201 load+serve campaign and the beta-vs-head bytes/cached_tokens oracle (blocker 4 / evidence gaps). Will push the branch onto this PR once the hardware receipts exist. @fivetide if you'd rather take any of the remaining items yourself, say which.

[Kaden-Schutt]

Pushed the maintainer repair directly onto this branch (fast-forward, 31682603f → b32b2321d): merge of beta 6208dec39 plus

• bf764df17 — blockers 1+2: real-writer flush after gen_start (macro + deepseek4 EP adapter); emit_aborted_terminal_after_abort helper in hipfire-engine, all seven Abort arms route through it; route-count assertion replaced by a flush-observing per-route test.
• e632b312d — blocker 3: DSpark rollback tests drive load_dspark_with_fault on the real deepseek-v4-flash-0731-dspark.mq2r (AfterLayer(0)/AfterHeadHelper/AfterMainProj/AfterGlobal), assert Err + free-VRAM recovery, retry clean, head GEMV to finite logits.
• b32b2321d — the first hipMalloc in a process on gfx1201/ROCm 7.15 permanently reserves 153,092,096 B driver-side (proved by a malloc/free ledger; a second fault load adds 0). Tests warm that by name before baselining; no slack change, rollback code untouched.

Hardware receipts (repair head vs beta 6208dec39)

gate	host	result
Oracle: Qwen3.8-27B, ornith A3B, LFM-1.2b — AR battery + chain, greedy	R9700 gfx1201	30/30 turns byte-identical content and cached_tokens
gen_start delivery	R9700	TTFB ≈0.3 ms vs 5.65 s prefill, both arms (flush proven)
Lifecycle: mid-stream cancel → follow-up; restart	R9700	exactly one abort terminal for the cancelled id, follow-up stop; post-restart clean. (No HTTP reset/unload route exists — 404.)
DSpark seam tests (real sidecar)	R9700	4/4 pass
Qwen3.8 AR + DFlash battery/chain	RX 7900 XTX	5/5 all; chain cached 0/284/493/674/939
ornith A3B q8 battery + chain	XTX	5/5, 5/5
DS4 deepseek-v4-flash-0731.mq2r + DSpark battery	Strix Halo gfx1151 (103 GB)	loads, sidecar paired, 5/5 stop on both arms; empties/sub-1 τ identical on beta (drafter quality, filed separately)

Not a #742 finding, recorded for honesty: lfm2.5-8b-a1b.mq4 on gfx1100 q8 greedy hits attractor 3/5 on beta and on this head alike.

Wire change to declare in the body: generate attempt_id: 0 was accepted by the beta parser and is now rejected (hipfire-daemon/src/main.rs:165-175, 2261-2275); first-party client handles it, direct clients using zero will not generate. Please add that to the PR description rather than "no compatibility weakening".

Remaining from the ledger (README.md:173-185) that this repair does not close: direct-Qwen embed/layer/norm/output fault boundaries, malformed-tool fixture, VL. I'd take those as follow-ups rather than block on them now that the oracle proves ordinary-path bytes unchanged. Re-review on request.

[Kaden-Schutt]

@fivetide — maintainer decision: we are finishing the device-mesh port to completion, and it is the only thing on our plate until it lands. Coordination for the G4 line stays on this PR; #666 remains the tracker of record. Beta → master promotion (#730) now waits on the composed G4 head, not just on this PR.

Proposed split, using your 2026-09-08 decomposition as the plan of record. Take or hand back any item by replying here; nothing is assigned to you without your say-so, and I'd rather you own the pieces you designed.

item	proposed owner	shape
this PR (G4.3–G4.6)	maintainer: re-review and merge to beta once the required jobs are green	as-is at 646f5a30c
G4.3 fault boundaries (embed / completed layer / final norm / output publication)	maintainer	stacked PR on beta; ignored GPU tests + gfx1201 run
G4.5 malformed-tool + open-think/length-cap fixtures	maintainer	harness fixtures + engine boundary tests
G4.7 Qwen35 attested reset/rollback (Single, then physical PP)	your call — it's your lifecycle design	one grouped PR; PP evidence runs on our 4× R9700
G4.8 vision/OCR lifecycle	maintainer (we have the Qwen3.8 vision sidecar + dots.ocr fixtures)	one grouped PR
G4.9 mesh TP/PP cancel + attested rollback in generate_mesh_carrier	your call	one grouped PR; physical TP/PP on 4× R9700 by us
G4.10 dense / LLaMA AR / batch-lane rollback	maintainer, unless you want dense	three PRs per your plan
composed G4 matrix + revert identity	maintainer runs on gfx1100 + gfx1201 + gfx1151	receipts posted on #666
G1–G3 final-head receipts	maintainer, on the composed head	tracker flips are yours
G5 re-cut (qwen3.6:35b-a3b sealed MoE)	after G4 accepted — owner TBD with you	fresh cut from beta; #677 stays provenance

Hardware we can commit: 4× Radeon AI PRO R9700 (gfx1201, mutually peer-accessible, RCCL), RX 7900 XTX (gfx1100), Strix Halo 8060S (gfx1151, 103 GB). That covers the physical TP/PP/EP rows that have been hardware-blocked.

Reply with what you want to keep; I'll start on the maintainer rows today.

[Kaden-Schutt]

Blockers 1–3 closed by the repair wave on this branch and proven on gfx1201/gfx1100/gfx1151 (receipts in the body); ordinary-path bytes and cached_tokens byte-identical to beta across Qwen dense, A3B and LFM; required CI green on 646f5a3. Merging to beta as the G4.3–G4.6 constituent — not G4 milestone acceptance; the open ledger rows are tracked in the plan comment above.

[fivetide]

I’ll take ownership of G4.7 Qwen35 attested reset/rollback and G4.9 mesh TP/PP cancellation plus attested rollback. Dense/G4.10 stays with the maintainer. After we provide the implementation and single-GPU evidence, the maintainer runs physical PP/TP on 4× R9700. We’ll update tracker flips only after composed receipts; this is not a completion claim.

[Kaden-Schutt]

@fivetide — thanks, and apologies for crossing wires: by the time your reply landed, G4.7 had already been executed here from your 2026-09-08 design (I'd said "your call" and then didn't wait — that's on me). Rather than throw it away, it's yours to review and own: branch g4/7-qwen35-reset (E0–E4 as five commits), and the composed head is g4/integration @ b8faa81d8 on origin. G4.9 is yours — please cut it from b8faa81d8; it consumes production_pp_fail_closed_rollback from G4.7 (common.rs), and the read-only plan we produced for G4.9 is attached below as design input, not a directive.

Composed G4 head b8faa81d8 = beta c88a1ba0d + the constituents below. All no-GPU gates pass on the composed head (leanup 21/21 — the G4.7 test-only daemon wiring was moved into hipfire_generate::common::RequestFaultGuard to stay under the 4816 ceiling; maps 44/44; ratchet-diff clean; 1303 tests across generate/runtime/arch-qwen35/engine/arch-deepseek4).

constituent	branch @ head	hardware evidence (all gfx1201 R9700 unless noted)
G4.3 direct-Qwen fault boundaries	g4/3-qwen35-fault-boundaries @ f082c547f	fault after embed / layer0 / last layer / final norm / output through the production run_staged_load transaction on qwen3.8-27b.mq4-xt: Err + free VRAM back to warmed baseline + clean retry + finite logits, 5/5; production path has no fault parameter (CPU test)
G4.5 parser finalization fixtures	g4/5-parser-fixtures @ 793c9c27f	latch tests green; serve_harness --mode session benchmarks/prompts/g45_parser_finalization.json: seed stop (1 cache store), malformed tool → length with no tool release, open-think → attested error, length-cap → length, reuse → stop; 0 cache stores on fail-closed turns, exactly one terminal each. Harness gained torn-stream error-terminal mapping (no Rust change)
G4.7 Qwen35 attested reset/rollback	g4/7-qwen35-reset @ f91f188e1	qwen35_reset_hw: fault-after-prefill + fault-after-first-decode → one attested error terminal each, clean retry byte-equal to fresh-daemon baseline, over 3 rounds — Single, PP2 (devices 0,1) and PP4 (0,1,2,3) all pass. serve→error flag list: empty (S5/P5 untouched). S6 serve_engine slot-reuse reset is production (spawned by daemon slots.rs) and remains an unpaired reset — listed as follow-up, not fixed here
G4.8 vision/OCR lifecycle	g4/8-vision-lifecycle @ 6e27f419f	vision_lifecycle_tests: dots.ocr matrix (cancel pre-prefill/decode, prefill/argmax/decode faults, spec fault, reuse, unload/reload VRAM) and Qwen3.8-VL matrix both green on gfx1201 (890 s); every exit → one terminal; text routes untouched
G4.10 dense / LLaMA AR / batch-lane	g4/10-dense-rollback @ af49a11c8	dense_rollback_gpu: LFM2 + LLaMA fault→attested terminal→retry envelope-equal→unload/reload clean; reset-phase error visible rolled_back=false; 3/3. DS4 rows are code-only (no single-GPU DS4)

Wire-visible deltas across all of it, for the record: rolled_back flips false→true where a real reset now happens; LLaMA/MiniMax/Cohere/Qwen2 loops gain cancellation polling where beta had none; batch lanes whose reset fails now retire with a visible unattested error instead of silently. No serve→error conversion anywhere.

Still open for the G4 milestone: G4.9 (yours), the composed-head matrix rerun after G4.9 lands (I'll run gfx1100/gfx1201/gfx1151 + the physical TP/PP rows on 4× R9700), the state oracle, and one revert identity. Then G1–G3 receipts on the same head so your tracker flips have something to point at.

[Kaden-Schutt]

Design input for G4.9 (read-only plan produced against beta c88a1ba0d; anchors may have shifted on b8faa81d8). Not a directive — your call on shape.

{
  "acceptance": "Parent runs: cargo test -p <crate> --locked for touched crates; ignored GPU tests on 2x and 4x R9700 gfx1201 with env below; rerun full G4 matrix after compose. No VALIDATION.md mesh route exists at c88a (grep empty) so no coherence-gate-style claim may promote; physical rows need pinned identities.",
  "anchor_root": "/home/kaden/.omp/wt/g4-10-dense-rollback (exact c88a1ba0d; hipfire-beta checkout is newer 6208dec39 — do NOT anchor there)",
  "beta_behavior_at_risk": [
    "Wire type/message unchanged at every site; only rolled_back flag may flip false->true (schema already carries it; verify no client gates on false).",
    "Bare-return sites (qwen.rs:440,557,663) gain a terminal where beta emits nothing — unreachable post-admission (admission pins EpArch), proven by new unit test on ep_serve_target.",
    "Trailer/budget/max_think closers (qwen.rs:5240,5420,5525): NO wire change (keep eprintln+break); reset addition only zeroes device state that beta leaves dirty (next-turn drift bug). Next-request parity test proves improvement.",
    "Jinja Plain fallback (qwen.rs:4710) retained; rollback added only on the hard-error return (qwen.rs:4699).",
    "No new cancellation poll points: PP polls at qwen.rs:5000,5092,5578; dense-TP polls per prefill chunk (qwen.rs:429) and per decode token (qwen.rs:536). Adequate.",
    "Refusals retained verbatim: tp>1&&pp>1 exclusive (main.rs:1654), EP+DFlash (main.rs:1659), EP+gemma drafter (main.rs:1664), PP+DFlash/CASK/PFlash gates (main.rs:1669-1693). No DFlash/CASK/PFlash/VL/spec on mesh."
  ],
  "coordinator_notes": [
    "G410DenseRollback acked: ping before touching ar.rs/dense.rs/batch lanes; consume terminal adapters without signature changes.",
    "G47ResetPlan: G4.7 lands production_pp_fail_closed_rollback(m,gpu,pp_gpus)->RollbackEpilogue in common.rs (host clear + pp recurrent + per-device graph invalidate + per-device sync), subsuming the bare macro at PP sites. Composer CALLS the helper, never the macro; macro stays until G4.7 lands.",
    "G43FaultBoundaries: model_load.rs gains StagedLoadFault + fault_error hook + run_staged_load_with_fault (additive; success path byte-identical). PP load reuses the hook where it composes; ping before touching model_load.rs."
  ],
  "evidence_hardware_2x_4x_r9700": [
    "Record every run: hipfire+daemon binary md5, prompt md5, model file md5, ROCm version, RCCL version, rocm-smi PCI bus IDs per GPU, gfx1201 arch-string match (preflight is exact-match, multi_gpu.rs), HIP_VISIBLE_DEVICES + ROCR_VISIBLE_DEVICES + hardware.devices, HIPFIRE_PP_LAYERS if set, HIPFIRE_TP_USE_RCCL if set.",
    "PP load: raw daemon JSONL load {params:{pp:2}} (pp is NOT a CLI flag; docs/multi-gpu.md) on qwen3.8-27b.mq4-xt (~/.hipfire/models); scripts/pp-gate.sh is the supporting gate script (explicitly not a VALIDATION minimum route).",
    "PP generate: prompt, decode N tokens, abort mid-decode via terminal control; inject HIPFIRE_MESH_FAULT=prefill|decode|reset; assert exactly one terminal, rolled_back=true, no cache publication, second request token-identical to fresh daemon, unload ok.",
    "PP multi-cycle VRAM: load->generate->unload x3; get_vram_info free bytes within 64MiB slack of baseline (DSPARK_VRAM_SLACK_BYTES model, arch.rs:3902); serialize with test mutex; warm first-alloc reservation (context value 153,092,096 B — VERIFY name/value at runtime via free-byte delta) by name.",
    "TP (dense): HIPFIRE_TP=2 serve of qwen3.8-27b.mq4-xt via load_model_ep dense-TP path; pre: tp_allreduce_smoke (HIP_VISIBLE_DEVICES=0,1[,2,3]) + qwen_dense_tp2_parity example (HIPFIRE_TP_PARITY_STEPS, HIPFIRE_TP_PARITY_PROMPT_FILE); inject decode fault; per-rank graph/KV reset; next-request parity; unload VRAM bound.",
    "TP/PP 4x: repeat with HIPFIRE_PP_LAYERS=a,b,c,d (sum=n_layers) and TP=4; record per-rank PCI + RCCL init (ensure_rccl lazy, multi_gpu.rs:917).",
    "Emulation may prove ONLY: staging order (TestWeightSource pattern, model_load.rs:501), terminal-latch unit tests, pp=1 macro-vs-helper equivalence. Never a multi-device row."
  ],
  "explicit_unknowns": [
    "No device-mesh-g1-g5-routes section in VALIDATION.md at c88a — claim-scoped mesh harness is undefined; parent designates. pp-gate.sh is explicitly not a minimum route.",
    "Per-device KV state beyond compact_offset on rollback (KvCache Multi-target internals) — confirm with arch owner that DN-zero + compact_offset=0 is complete; else extend G4.7 helper, not G4.9.",
    "Device-sampler finite-logit validation on PP path (no host download per token by design); logit injection covers forward-Err, not NaN logits. Host download per token is explicitly NOT proposed (perf).",
    "153,092,096 B first-alloc reservation value/name — verify at runtime via get_vram_info delta on parent hardware before freezing test constants.",
    "load_model_ep staging guard covers dense-TP load faults already (main.rs:969); verify, do not duplicate."
  ],
  "files_changed": [
    "crates/hipfire-generate/src/qwen.rs (generate_mesh_carrier new; PP sites 4699/4971/5123/5240/5420/5525/5578; dense-TP sites 333-736; ep_serve_target 75-110 untouched)",
    "crates/hipfire-generate/src/common.rs (CONSUME G4.7 production_pp_fail_closed_rollback + ep_reset_after_abort; add only reset_mesh_request_state dispatcher; fail_closed_reset_target_and_spec 1142 + reset_qwen35_recurrent 772 owned by G4.7/G4.10 — no edits)",
    "crates/hipfire-loader/src/carriers.rs (load_qwen35_pp 240-361 staging only; admit_topology 387-400 untouched)",
    "crates/hipfire-runtime/src/model_load.rs (reuse G43 run_staged_load_with_fault; ping G43 first)",
    "crates/hipfire-generate/src/ar.rs + hipfire-engine/src/terminal.rs (CONSUME ONLY: adapters 1322, emit fns 1367-1401, await 1291, check_abort 1159 — owned by G4.10, message first)",
    "NO changes: device_mesh.rs, multi_gpu.rs Gpus/collective signatures, daemon admission, dispatch routing, no new crates"
  ],
  "key_findings_c88a": [
    "generate_mesh_carrier does NOT exist (tree-wide grep empty) — G4.9 creates it as a thin lifecycle wrapper in qwen.rs, not a new loop (per #666 rule 2: no port of branch generate/mesh.rs).",
    "PP generate = generate_multi (qwen.rs:4411-5589), dispatched at ar.rs:2411. Cancellation polling EXISTS (qwen.rs:5000 post-prefill, 5092 per-decode, 5578 commit handshake). Gap is attestation: error sites (4971 prefill, 5123 decode, 4699 jinja-hard-error) use infallible reset_pp_uncommitted_state! macro (4864; ignores memset errors, no device sync/graphs, no rolled_back flag) + emit_generation_error(rolled_back=false). Trailer closers (5240/5420/5525) swallow via eprintln.",
    "PP reset owner already pp-aware: reset_qwen35_recurrent (common.rs:772-824, G4.7-owned) does per-device bind+memset with error accumulation. fail_closed_reset_target_and_spec (common.rs:1142) does single-gpu-only sync (1242) + single-gpu graph invalidate (1236). G4.7's forthcoming production_pp_fail_closed_rollback closes exactly this gap — G4.9 consumes it.",
    "Dense-TP generate = ep_serve_qwen35_dense_tp (qwen.rs:333-736) via generate_ep dispatch (qwen.rs:263) on EpArch::Qwen35DenseTp (ep_serve_target 75-110). Has polling (429/536), pre-request per-rank reset+graph-invalidate (363-394), commit handshake on success (ep_emit_done 783). Gaps: ALL error sites (474/499/511/575/610/669/681/702) emit_active_route_error(rolled_back=false) with NO reset; bare returns with NO terminal at 440/557/663. Reusable as-is: ep_reset_after_abort covers DenseTp per-rank bind/reset/sync (830-849) + ep_emit_abort (886).",
    "PP load = load_qwen35_pp (carriers.rs:240-361) is NON-transactional: bare ?-chain, late KV/DN/scratch failure leaks weights (no free_gpu_multi). Staging pattern to reuse: StagedLoadOps/run_staged_load (runtime/model_load.rs:151/190) + G43's run_staged_load_with_fault; DSpark seam-test model arch.rs:3927-4115 (env HIPFIRE_DSPARK_FIXTURE, warmup 1MiB const 3922, VRAM mutex 3909, slack 64MiB 3902).",
    "Carrier-owned mesh state: Qwen35Bundle.pp_scratch_set (carrier.rs:29, freed via free_gpu_multi NOT ArchModel::free_gpu) + LoadedModel.pp_gpus/pp_dn_la_to_device (lib.rs:935-937, skeleton_pp 1304) + per-device KV/DN/weights; PP unload arm (lib.rs:~3995-4025) already frees all four multi + per-device invalidate/drain. PP bypasses hipGraph capture (ep_batch.rs:4035 comment) so 'graph reset' = per-device invalidate_graph_state + Gpus tp_graph_signals; RCCL lazy via ensure_rccl (multi_gpu.rs:917, opt-out HIPFIRE_TP_USE_RCCL=0).",
    "Daemon naming trap: params.tp = EXPERT-parallel (main.rs:1646-1648), NOT tensor-parallel. TP lifecycle in G4.9 scope = dense-TP via EP serve path (forward_prefill/forward_scratch_dense_tp, forward.rs:4259/4121; load_weights_dense_tp_rank, load.rs:3406). tp>1&&pp>1 mutually exclusive (main.rs:1654)."
  ],
  "not_doing": [
    "No TP+PP combo, no DFlash/CASK/PFlash/VL/spec/tool paths on mesh, no DeviceMesh/Gpus/collective-schedule changes, no daemon admission or 

[fivetide]

Thanks, Kaden. Our adversarial review of f91f188 / b8faa81 found three G4.7 gaps before we can own it: (1) PP main decode publishes a token before forward; the injected first-decode seam is pre-forward, so a real post-forward failure can publish then error; (2) PP max-think close, budget nudge, and trailer swallow forward errors and can continue toward done; (3) context-full/Jinja pre-prefill reset failure uses single-GPU rollback rather than all-PP-device attestation. We will correct these atop b8faa81, then implement G4.9 from that corrected composition. Please defer the physical rerun until that branch is posted.

[fivetide]

G4.9 follow-up is ready in PR #747: #747 (amended head 5f5d7fa07971656277ae7ebe085f9741ffdf0604). Independent amended-diff audit is clean; reset ownership, duplicate-context rendering, and the production-seam test blocker are closed. The branch is ready for composition; physical PP2/PP4 and dense TP2/TP4 R9700 validation remains required, including no token/committed/done/aborted after injected failure and next-request byte parity.