[fusheng-ji]

RFC #420 stack 10/10 (base test-h3, merge bottom-up): #483 → #484 → #485 → #486 → #487 → #488 → #489 → #493 → #494 → #539. This PR merges after #494; until the lower PRs merge, GitHub also shows their commits. This PR's own diff.

Evidence location: the measurement outputs of this PR (JSON, figures, audit files) are not part of its diff. They are on the fork's pr_assets branch under pr539/ (links in this description are fixed to that commit). Paths such as reports/experiments/... below name where a runner writes its output; the committed copies are under pr539/reports/experiments/ on that branch (h3-full-model-b200/, h3-one-block-b200/) and the H100 files under pr539/h100-full-model/.

What

RFC #420 row ws1_one_h3_block: one real H3 block, forward and backward, with every intermediate and a first-drift report.

• The block. Block 0 of MiniMax-H3@42ed227 runs as a 17-node graph, from the AdaLN projection to the second gated residual. It runs on packed FL2VA layouts whose position_ids and tags are bitwise equal to the pinned diffusers pipeline's build_packed_sequence; the hashes are pinned in a CPU test.
• Three executions per node. Each node runs as:
  • the diffusers replay (op for op, no diffusers import);
  • the RL-Kernel binding, both chained and isolated (fed diffusers' inputs);
  • an FP32 golden.
• Ownership. Each node names the RFC row that owns its arithmetic, and whether it is bound to that row's operator, an interim deterministic operator, or the provider replay.
• Forward report, per node:
  • repeat and batch-row bitwise checks;
  • chained and isolated agreement with diffusers (including the first differing packed row);
  • candidate and diffusers error against the golden;
  • the node's sha256;
  • first_{,isolated_,repeat_,batch_}drift.
• Backward report:
  • gradients of the block input, temb and all 12 block parameters (repeat, error vs golden);
  • the gradient reaching every node (repeat, batch row), with first_grad_{repeat,batch}_drift.
• Weights. Block-0 attention/FFN weights are pinned in a new manifest section (block_tensors). They are extracted by prepare_h3_weights.py --block from the same shard, so the existing conditioning file is unchanged.

Bindings

Node	Owning row	Bound to
adaln_projection	adaln_projection_3mod	row (#485), via the registry
norm1, norm2	h3_rmsnorm	row (#487), via the registry
residual_attn, residual_mlp	adaln_gate_residual	row (#488), via the registry
q/k/v_proj, o_proj, ffn_gate_up, ffn_down	h3_qkv_gemm, h3_attention_o_gemm, h3_ffn_gate_up_gemm, h3_ffn_down_gemm	interim: Triton FP32-accumulate GEMM, one BF16 rounding
q_norm, k_norm	h3_qk_rmsnorm_d128	interim: #487's plain H3RMSNormCudaOp on the 128-wide heads (bitwise equal to nn.RMSNorm)
rope_q, rope_k	h3_mm_rope_3axis_partial	reference: the provider replay, since no MM-RoPE kernel exists yet
attention	h3_full_attention	interim: DeterministicAttentionOp(causal=False)
swiglu	h3_swiglu	interim: SwiGLUCudaOp

When an owning row lands, swapping its node's candidate callable is the whole integration, and the report entries become that operator's block-level evidence.

Full-model and generation qualification (added 2026-10-12; numerical acceptance passed, three-seed generation quality not passed, see Status). This part covers the WS1 closeout rows ws1_full_h3_transformer and ws1_dual_trajectory, which are not claimed yet; it reuses the block graph above for every layer.

• tools.validation.models.h3_full_model runs the real 50 transformer blocks, both text-refiner layers and the input/conditioning/output components. Each candidate block binds its own parameters to the block-0 graph, and the row/interim/reference labels are kept. Every parameter and input gradient is checked.
• A reference pass (independent FP32 and BF16 provider runs) freezes per-tensor error bounds with a committed formula: 764 tensor bounds in examples/models/minimax_h3/full-model-thresholds.json, and a 1,635-file SHA-256 reference inventory.
• The candidate gate builds its own H3 extension, verifies every reference and checkpoint hash, then runs the block-0 golden tests (no skips), full-model forward/backward with every candidate node required to run (CUDA fallback rejected) and first drift ordered by real forward/backward execution. Only a passing numerical comparison allows the preflight and the three-seed candidate generation.
• Generation follows generation-contract.json: the pinned public reference image and prompt, seeds 42/43/44, 124 frames, 1344×768, 24 fps with joint 32 kHz audio. The comparator checks identical initial noise and shapes and reports PSNR/SSIM, temporal RMSE and audio differences.
• Candidate generation takes about 20 hours per seed, so it runs as one resumable task per seed: saved sampling steps are replayed after a preemption instead of recomputed, with every replayed call's inputs checked against their saved digest (h3_generation_resume.py; recomputing the last saved step and requiring a bitwise match is opt-in; see Status).

Prior art & reuse decision

This is a validation closeout, not a new kernel; no operator is written here.

• diffusers. MiniMaxH3TransformerBlock is the executable reference. It is replayed op for op as the provider (h3_provider), which pins its dtypes and op order without importing diffusers.
• [ws1]: WS1 Full Qwen3-8B Dense Train-Inference Closeout #315 Qwen3-8B chain gate (rl_engine/validation/models/chain_gate.py). Its method is reused: per-node digests and first drift in graph order. Its graph is Qwen3's (causal, no AdaLN, Qwen shapes), so the H3 block is built on the h3_chain stage pattern of [WS1][CUDA][MiniMax-H3] timestep_sinusoid_h3: FP32 timestep features + shared H3 harness (RFC #420 stack 1/10) #483.
• Interim operators are reused from RL-Kernel. The one decision taken here is the GEMM entry point. TritonDetGemmOp.__call__ reduces K as a tree with BF16 nodes; its FP32-accumulate kernel is equally deterministic and row-invariant, matches cuBLAS's accuracy, and is 9–17× faster (B200, error vs FP64):

Shape	Tree path	FP32 K loop → BF16	cuBLAS	Tree time	FP32 K loop time
264 × 5376 → 7168	7.0e-3	2.4e-3	2.4e-3	10.4 ms	1.2 ms
925 × 14336 → 5376	7.5e-3	2.6e-3	2.6e-3	47 ms	2.7 ms

Direct bitwise comparison

The H3 block is composed of the WS1 rows below it, and their reuse candidates are compared byte for byte with tools/validation/models/h3_bitwise_reuse.py (committed in #483, merged down the stack). It runs every candidate that h3_prior_art.candidates() defines on identical inputs and compares every output and leaf gradient with diffusers' path and with the rl-kernel op. Each row's report, written from a clean tree on its own branch: #483 timestep_sinusoid; #484 timestep_mlp; #485 adaln_projection; #486 adaln_row_gather; #487 norm_modulate; #488 gate_residual; #489 final_adaln_out.

Results

Block-0 replay, generated from a clean tree at 41e4293 on an otherwise idle B200 (torch 2.13.0+cu130), with tools/validation/models/h3_block_replay.py --layouts tiny,small,medium: report.json.

S	First drift (chained / isolated)	Repeat or batch drift, any node or gradient	Output err: RL-Kernel / diffusers	Fwd ms	Fwd+bwd ms
264	adaln_projection / adaln_projection	none	5.21e-3 / 5.21e-3	7.4 vs 0.61	22.7 vs 2.8
925	adaln_projection / adaln_projection	none	7.87e-3 / 7.87e-3	30.1 vs 1.12	81.7 vs 5.0
3160	adaln_projection / adaln_projection	none	8.97e-3 / 8.97e-3	225 vs 3.8	577 vs 11.5

• Isolated bitwise promises. Every node that promises it is bitwise equal to diffusers on diffusers' inputs at every size: norm1/norm2, q_norm/k_norm, RoPE, and both gated residuals. The first drift is the first reduction whose tree differs from cuBLAS.
• Forward accuracy. Per-node error vs the FP32 golden is 0.93–1.20× diffusers'. The largest is the interim attention at S = 3160.
• Gradient accuracy:
  • The candidate is up to 7× more accurate than diffusers on adaln_proj.weight, adaln_proj.bias and temb. diffusers' index_select backward accumulates in BF16 atomics; the row operators use FP32 segment sums.
  • The candidate is within 1.16× on the other leaves, except attn.norm_q.weight at S = 3160 (9.5e-3 vs 5.5e-3). At S = 925 the same leaf is 2× more accurate than diffusers, so the 128-element gradient's error follows BF16 rounding, not one side.
• Control H-C3, reported, not promised. Permuting the packed rows with their positions and indices changes 79 / 450 / 2199 output rows (max 8). Attention sums keys in packed order; settling this belongs to h3_full_attention.
• Speed. The 12–60× forward cost is the interim GEMMs and attention (FP32 scores materialised), not the row operators.

Full-model qualification

Gates of candidate.sbatch at c249a735 on one B200 (torch 2.9.1+cu130) against the regenerated reference, from a clean checkout; generation from clean checkouts at 4792c8a9 / 2451ae5c (same model code, see Status). Evidence: README, acceptance, comparison, generation quality, provider variants, first-step replay, provider spread.

Gate	Result
Block-0 golden tests	9 passed
Full-model forward/backward (50 blocks, both refiners, boundaries), 764 tensors, provider-ensemble rule	passed, 0 failing; worst tensor at 0.55 of its bound (block 39 adaln_proj.linear.weight)
Final outputs vs the FP32 reference, relative_l2	video 0.0063 (0.35 of its bound), audio 0.0218 (0.39)
The original single-configuration bounds (kept as a record)	54 of 764 fail: 52 on max_abs only, 2 also on relative_l2
Generation preflight (128×128, one step)	complete
Three-seed generation, 124 frames 1344×768, 49 steps	complete
Generation quality vs the provider, bounds fixed before generation	not passed: seed 44 passes, seeds 42 and 43 do not

Generation quality per seed (bounds: PSNR ≥ 30 dB, SSIM ≥ 0.99, temporal RMSE ≤ 0.01, audio relative L2 ≤ 0.05; a failing value is bold):

seed	PSNR (dB)	SSIM	temporal RMSE	audio relative L2	passed
42	37.58	0.9917	0.00991	0.0807	no
43	32.64	0.9757	0.01056	0.0762	no
44	38.75	0.9945	0.00734	0.0456	yes

The same bounds applied to the provider with one SDPA backend forced on the transformer (same seeds, inputs and conditioning):

provider variant	seeds passing	PSNR (dB), seeds 42 / 43 / 44	audio relative L2, seeds 42 / 43 / 44
flash attention	0 of 3	39.08 / 23.98 / 39.33	0.054 / 0.264 / 0.056
memory-efficient attention	0 of 3	39.43 / 30.22 / 42.59	0.094 / 0.135 / 0.116
candidate	1 of 3	37.58 / 32.64 / 38.75	0.081 / 0.076 / 0.046

First denoising step replayed from its saved packed inputs (nothing has fed back yet), relative L2 of the video output against the provider's saved step (replaying the provider is bitwise equal to it):

configuration (seed 43 unless noted)	video relative L2
candidate (seeds 42 / 43 / 44)	0.0285 / 0.0212 / 0.0274
provider, flash / memory-efficient attention	0.0219 / 0.0218
provider, cuBLASLt	0 (bitwise equal at this step)
candidate with attention / the six GEMMs / SwiGLU / the AdaLN projection run as the provider	0.0225 / 0.0217 / 0.0198 / 0.0193
candidate with Q/K norm, the two norms or the residuals run as the provider	0.0212 (unchanged by the swap)
candidate with all interim operators / every node run as the provider	0.0192 / 0

Status

• Numerical acceptance of the full model: passed (gates at c249a735 against the regenerated reference).
  • Block-0 golden tests: 9 passed. Full-model forward/backward over all 50 blocks, both refiners and the boundaries: all 764 tensors within the provider-ensemble rule (relative_l2 at most 2.6 × the largest provider error + floor); worst tensor at 0.55 of its bound. The final video and audio outputs are at 0.35 and 0.39 of theirs. The numbers are identical to those of the first reference run, whose files are kept next to them. See Results.
• Generation quality: not passed. Three seeds were generated (124 frames, 1344×768, 49 steps). Against the provider, with the bounds fixed before candidate generation (PSNR ≥ 30 dB, SSIM ≥ 0.99, temporal RMSE ≤ 0.01, audio relative L2 ≤ 0.05), seed 44 passes and seeds 42 and 43 do not: audio relative L2 is 0.081 and 0.076 (bound 0.05), and seed 43 also misses SSIM (0.9757) and temporal RMSE (0.01056). The bounds are unchanged.
• What the comparison says about those bounds.
  • The bounds had only been checked against a bitwise-identical repeat of the provider (PSNR 300 dB, zero error). The provider with one SDPA backend forced on the transformer, run through the same 49 steps with the same seeds, inputs and conditioning, fails them too: flash attention 0 of 3 seeds, memory-efficient attention 0 of 3, and audio relative L2 exceeds 0.05 on every seed of both (0.054 to 0.264). Seed 43 is the hardest for every configuration (PSNR 23.98 with flash attention, 30.22 with memory-efficient attention, 32.64 for the candidate).
  • The first denoising step, replayed from its saved inputs, shows the same: the candidate differs from the provider's step by 0.021 to 0.029 relative L2 on video, and the provider with flash or memory-efficient attention differs from its own default by 0.022. Running any one group of block nodes (attention, the six GEMMs, SwiGLU, the AdaLN projection) as the provider's leaves 0.019 to 0.023, replacing the norms or the residuals changes nothing at this step (consistent with those nodes being bitwise to the provider's), and only running every node as the provider's gives zero. No single node accounts for it. It is the BF16 noise floor that any change of kernel produces, which the 50 sampling steps then amplify.
  • This is not a claim that the candidate is better than the provider variants: on seeds 42 and 44 its PSNR and temporal RMSE are slightly worse than both variants' (seed 42 PSNR 37.58 against 39.08 and 39.43), on seed 43 better, and it passes one seed where they pass none.
  • The generation bounds were not changed and the candidate is not accepted on generation quality. A rule that tolerates a change of kernel, like the one for the numerical gates, would be a second rule designed after seeing a failure; it is left to the reviewers.
• The reference was regenerated. The first reference run's artifacts (about 200 GB) were deleted to save disk space, so they were regenerated with the original commands at the same source commit (2681cadb). 1,614 of the 1,635 pinned files are byte-identical to the first run's; the others are the eight report.json files, the thresholds file that embeds their hashes and twelve video.mp4 / audio.wav containers, which also differ between two runs of one job. The calibrated bounds and the ensemble rule re-derived from the regenerated reference are identical in every numeric field; the manifest, thresholds and rule are re-pinned to it (c249a735) and the first run's hashes are in the model README.
• How the acceptance rule came about.
  • The bounds frozen from the reference run (1.5 × one provider pass + floor) failed 54 of 764 tensors for the candidate. The same bounds also fail diffusers runs that differ only in attention backend or BLAS library: flash attention 36 tensors on max_abs (1 on relative_l2), cuBLASLt 38 (3), memory-efficient attention 66 (9), while the default configuration they were calibrated on fails none.
  • A diagnostic that ran the full model with chosen block nodes replaced by their diffusers replay (attention, the six GEMMs, SwiGLU, Q/K norm, the two norms, the residuals, the AdaLN projection, all interim operators, all nodes) found no single group responsible: replacing any one leaves 3 to 16 of the 16 clearest outlier tensors above 1.5× the providers' maximum, and replacing every node reproduces the default diffusers run exactly on all 764 tensors. These hybrids are diagnostic only and not part of the committed evidence.
  • The acceptance rule is therefore calibrated on four distinct provider configurations only, with no candidate data: relative_l2 ≤ k × the largest provider error + floor, with k = 2.6 the smallest factor at which every provider configuration, held out in turn, passes against the other three. max_abs, which is set by one element, is reported but not gated: held-out providers exceed the others' maximum by up to 4.1× on it.
  • The rule was designed after the candidate had failed the earlier bounds. That is why the comparison against the old bounds is kept in the evidence next to the new decision.
• Block-0 gradient bound. The block-0 backward test compared each gradient's relative error with 1.5 × diffusers' + 5e-3. For a BF16 leaf that number is set by one rounded element and moves in whole ulps, and it failed on one seed with torch 2.13 and another with torch 2.9.1 although no row operator moves it. A BF16 leaf now also passes at most 6 ulps above diffusers'. Sweeps over the tiny and small layouts × 10 seeds measured at most +1.86 ulps (torch 2.13) and +4.15 (torch 2.9.1), and 34 single-node swaps at most +5.15 (reports).
• Block-0 replay. The evidence at 41e4293 / caaf754 (Results) has no known failures.
• CPU checks. Lint, unit tests and docs pass at 58e08219. The DCO check reports a failure because it cannot read more than 250 commits; every non-merge commit on this branch (284 commits in all, 130 of them non-merge) carries Signed-off-by (checked with git log).
• Fixed: the flaky test_h3_adaln_modulation.py::TestFusedModulation::test_backward_keeps_table_gradient_fp32 in [WS1][CUDA][MiniMax-H3] adaln_row_gather: fused six-way AdaLN row gather (RFC #420 stack 4/10) #486 (62f8009e), merged here. Its golden rounded an FP64 SiLU where the kernel rounds the FP32 SiLU, and temb was unseeded; the kernel was deterministic.

History

• 2026-10-18: investigation of the generation quality (8324ec69, 8493d5ba, 9951ddab, ebd0512b).
  • h3_first_step_diagnostic.py replays the first denoising step from its saved packed inputs under a given configuration (the provider, optionally with one forced kernel; the candidate; the candidate with chosen block nodes run as the provider's). Replaying the provider default reproduces the saved step bitwise on all three seeds, which validates the replay.
  • h3_full_generation --provider-variant generates the provider's videos with one SDPA backend forced. The first attempt forced it around the whole pipeline, so the Qwen3-VL text encoder crashed under memory-efficient attention and would have had its text embeddings changed under flash attention; it now applies to the transformer's forward only, and the first attempt's outputs were discarded.
  • Results are in Results and Status; the evidence is in h3-full-model-b200/ on pr_assets.
• 2026-10-17: generation complete and compared. All three seeds reached step 49 after 7 to 11 attempts each (the cluster preempted the tasks many times). The first quality comparison job failed on import because generation-compare.sbatch did not put the dependency overlay (scikit-image) on PYTHONPATH; fixed in fcd4ed27 and rerun.
• 2026-10-16: generation without recomputation, tied to the model paths, reference regenerated (1a7a2fc1, 2451ae5c, c249a735).
  • Late on 2026-10-16 the cluster preempted tasks every 16 to 71 minutes, and each resume recomputed the last saved step (about 25 minutes) plus about 5 minutes of loading, so an attempt shorter than about 55 minutes made no progress. Resume now replays every saved step and computes only what is left; recomputing the last saved step and requiring a bitwise match is --verify-last-step (opt-in; it was checked across nodes on the first run).
  • The validated execution trace is saved after an attempt's first real step, so an attempt that only decodes still reports a complete one. A run is tied to the git trees of the model paths (MODEL_PATHS), not a commit, so a commit that changes only orchestration code can continue it; h3_generation_adopt.py moves an older commit-based run over, only if those paths are byte-identical (checked for all seeds before any is changed). generation.sbatch accepts the numerical gates by the same identity. The 57 steps saved at 4792c8a9 (18, 19 and 20 for seeds 42, 43 and 44) were adopted this way, which is why seeds 42 to 44 carry steps computed under two commits with identical model paths.
  • Real-pipeline check at 1a7a2fc1 (preflight size): a resume with nothing left to compute and the opt-in verification both reproduce the uninterrupted run byte for byte.
  • The reference was regenerated and the manifest, thresholds and ensemble rule re-pinned (c249a735); see Status.
• 2026-10-16: resumable generation (db8c69c7, 026fd60e, 4792c8a9, f4e9a37f; revised below).
  • The first full candidate run was preempted 20 times: the longest attempt ran 12 hours, the furthest generation reached step 27 of 49 of the first seed, and every restart began from scratch; three seeds at about 20 hours each (about 1500 s per sampling step) also cannot fit one 24-hour job.
  • h3_full_generation takes --seeds and --resume. Each saved step has a sidecar with the digest of the transformer's inputs and of the saved file, written atomically. On resume the saved transformer calls are replayed from disk and the pipeline's own scheduler turns them into the next latents; every replayed call's input digest is checked. A run directory is tied to the commit, contract, seed and settings it started with (revised below).
  • Job scripts: candidate.sbatch ends after the numerical gates and the preflight; generation.sbatch runs one seed per array task and refuses to start unless those gates passed; generation-compare.sbatch compares afterwards.
  • Checked on the real pipeline at preflight size (128×128, 6 sampling steps): one uninterrupted run against runs preempted and resumed. The step files, the initial inputs, the video and the audio (8 files) are byte-identical. The first attempt of this check failed because the replay wrapper hid the transformer's forward signature from the pipeline (fixed in 4792c8a9, with two tests that fail without the fix); f4e9a37f fixes a mypy error that CI found in the new module.
• 2026-10-14: provider-ensemble acceptance rule (3fc8f93a, c8ac5b19, 9b1e3ee7, 6e114f9c, 48f60361): --provider-variant and --provider-nodes on the full-model runner, h3_full_provider_spread.py, h3_full_ensemble_rule.py and the frozen full-model-ensemble-rule.json. A candidate run at a7d3f9d0 completed the full-model forward/backward and failed 54 of 764 single-configuration bounds, so generation did not start; see Status for what followed.
• 2026-10-13: block-0 gradient bound in ulps (5e5df82f, a7d3f9d0), after a candidate run at 98b07a3f stopped at the block-0 gate (8 passed, 1 failed: the attn.norm_q.weight gradient's relative error 0.03398705939217804 against the bound 0.03044327425907444, provider 0.016962182839382958). With identical inputs and upstream gradients the isolated Q/K norm errors of candidate and provider are equal (Q 0.0016656962, K 0.0029305728), so the difference comes from upstream.
• Reference run (clean source 2681cadb): FP32/BF16 provider passes over all 50 blocks, both refiners and the boundaries, with 638 parameter-gradient artifacts per precision. Reference generation and repeats passed for seeds 42/43/44 at 124 frames, 1344×768, 24 fps and 32 kHz audio, and all 49 predicted-velocity steps matched their repeated provider controls. These are provider results.
• 2026-10-13: fast-math guard (from the #483 review). KERNEL_ALIGN_USE_FAST_MATH=1 adds --use_fast_math to the whole _C build, which approximates expf/sinf/cosf/rsqrtf/division and flushes denormals. Before the fix a fast-math build of the top of the stack failed 33 H3 tests (the timestep_sinusoid_h3 and h3_rmsnorm bitwise tests and the chain/block tests built on them; det_linear, adaln_row_gather and gate_residual still passed). Every csrc/cuda/h3 source is now in H3_BITWISE_SOURCES and left out of fast-math builds, the bindings are gated on KERNEL_ALIGN_WITH_H3_KERNELS, and dispatch falls back to the PyTorch references. On this branch: merge 4ba3a8fb and 9cb10d03 (block dispatch-check skip). Results for both builds are under Tests.
• 2026-10-13: site identifiers removed from the full-model kit (0348e37b). The .sbatch scripts keep only site-neutral resource requests (site-specific scheduler options are given at submission), the reference manifest is reference-manifest.json without a job ID (its 1,635 file hashes are unchanged, so the reference evidence still verifies), and generation-inputs.json records the source suite relative to its pinned revision.
• 2026-10-13: documentation repair (9317a5e1). 14 report links outside the MkDocs tree now point to immutable GitHub URLs at 68e472b7; the strict docs build passes. No evidence or numbers changed.
• 2026-10-12: full-model and generation qualification added (ae620486 onward; see What). An earlier reference attempt completed its backward stages but failed generation because torchvision/the video processor was missing; the processor repair and a matching torchvision overlay fixed the rerun. Earlier local expanded CPU results (149 passed, 358 hardware skips, three subprocess-startup timeouts) are historical, not an all-pass claim.
• 2026-10-12: evidence directory migration (03cd824b). The one-block report.json and figure.png moved to reports/experiments/h3-one-block-b200/ (now on pr_assets: h3-one-block-b200/) with identical SHA-256; the report keeps its 41e4293 measurement revision. Docs, the plotting example and the layout audit map were updated; no GPU numbers were rerun.
• Earlier test runs, superseded by Tests below: 2026-10-11 at caaf754, tests/models/minimax_h3 on 2× B200: 501 passed, 5 skipped; with the fast-math guard (2× B200, passed / skipped / failed): default 545 / 7 / 0, fast-math 196 / 356 / 0; after the flaky-test fix at 27bb0295: default 550 passed / 7 skipped, fast-math 201 / 356, 0 failed; uniform cluster run at 3bd2300a (8× B200): default 554 / 3 / 0, fast-math 201 / 356 / 0.

Tests

At ebd0512b (the head when this was measured; 58e08219 only moves the measurement outputs out of the PR and edits docs/operators/), 2× B200, pinned MiniMax-H3@42ed227 weights including block 0; each build compiled from a clean checkout of that commit:

python tools/weights/prepare_h3_weights.py --out <dir> --block && export RL_KERNEL_H3_WEIGHTS=<dir>
python setup.py build_ext --inplace                                  # default build
python -m pytest tests/models/minimax_h3 tests/build tests/runtime/test_dispatch.py -q
KERNEL_ALIGN_USE_FAST_MATH=1 python setup.py build_ext --inplace     # fast-math build, same pytest

Build	H3 kernels in _C	passed / skipped / failed
default	yes	572 / 7 / 0
KERNEL_ALIGN_USE_FAST_MATH=1	no (PyTorch reference fallback)	223 / 356 / 0

Default-build skips (7): 4 need 4 or 8 GPUs (the tensor- and sequence-parallel NCCL tests, on a 2-GPU host); 3 depend on the environment only (matplotlib, SGLang's JIT, CI-only extension enforcement).

Fast-math regression tests: tests/build/test_build_platform_collectives.py::test_cuda_build_compiles_h3_kernels_without_fast_math, ::test_fast_math_build_excludes_bitwise_h3_kernels, tests/runtime/test_dispatch.py::test_h3_timestep_falls_back_when_fast_math_build_omits_kernel, ::test_h3_ops_prefer_cuda_and_fall_back_to_pytorch_reference.

tests/models/minimax_h3/test_h3_one_block.py checks the block-0 replay: on CPU, that the layouts are pinned to diffusers, the graph is well formed and MM-RoPE rotates 96 of 128 channels; on GPU forward (S = 264, 925), that every node repeats and is batch-row bitwise, promised nodes are isolated-bitwise and node and output errors are at most 1.25 × diffusers' + 1e-3; on GPU backward (S = 264), that every leaf repeats, dX is batch-row bitwise, no gradient drifts and leaf errors are at most 1.5 × diffusers' + 5e-3, or, for a BF16 leaf, at most 6 ulps above diffusers'.

tests/models/minimax_h3/test_h3_generation_resume.py (22 CPU tests, included in the counts above) drives a fake denoising loop through the same resume code: a run preempted once or repeatedly equals an uninterrupted one, nothing is recomputed by default, the opt-in verification catches a drifting recompute, other identities, corrupted files and signature changes are rejected, and adopting a commit-based run is tested against a real temporary git repository. The real-pipeline checks are described in History.

On this PR the GitHub checks are DCO (see Status), CodeRabbit, lint, the CPU unit tests and docs.

Scope / Known limitations

• ROCm. CUDA only. The interim attention and Triton GEMM build for ROCm, but the block has not been run on AMD hardware.
• Interim bindings. 11 of 17 nodes are interim or reference. Their speed and rounding are not their owning rows' final ones, and RoPE carries no RL-Kernel evidence until h3_mm_rope_3axis_partial lands.
• Stand-in inputs. The block input is a seeded stand-in (the input projections are other rows), and temb is the provider's (the conditioning chain is [WS1][CUDA][MiniMax-H3] timestep_sinusoid_h3: FP32 timestep features + shared H3 harness (RFC #420 stack 1/10) #483–[WS1][CUDA][MiniMax-H3] adaln_row_gather: fused six-way AdaLN row gather (RFC #420 stack 4/10) #486).
• Block-0 replay inputs. The 17-node replay runs block 0 only; the 50-block, refiner and generation gates are the full-model qualification below, whose results are in Status below.
• Full-model qualification. Boundary components (input projections, conditioning, output heads) stay provider/reference. Numerical acceptance has passed; the three-seed generation quality has not (1 of 3 seeds, against bounds that the provider with another attention kernel does not pass either; see Status). Candidate generation is more than 50× slower than the provider's (about 20 hours per seed, against 1 h 7 min for the whole provider reference run with all three seeds); the cause is probably the interim attention, which computes full FP32 attention scores (not profiled), and h3_full_attention is the row that would remove it. The generation environment needs torchvision matching PyTorch, and Qwen3VLProcessor needs its video processor even for image conditioning.
• Acceptance rules. The full-model numerical gate is relative_l2 only (max_abs is reported); it and the block-0 ulp allowance rest on measured provider spread and were designed after the candidate had failed the earlier bounds. The generation quality bounds were not changed (see Status).
• Reference artifacts. The first reference run's artifacts were deleted and regenerated; the pinned inventory is re-pinned to the regenerated files, 21 of 1,635 of which differ from the first run's (metadata and media containers).

Results (H100 / Hopper)

Measured on 4 x NVIDIA H100 80GB (sm90), driver 580, CUDA 13.0.88, torch 2.13.0+cu130, pinned MiniMax-H3@42ed227, clean tree at 58e08219. Evidence files live on the pr_assets branch under pr539/reports/experiments/*-h100/.

Tests (pytest tests/models/minimax_h3 tests/build tests/runtime/test_dispatch.py)

• default build: 575 passed, 4 skipped in 219.42s (0:03:39)
• KERNEL_ALIGN_USE_FAST_MATH=1 build: 224 passed, 355 skipped in 49.38s (the fast-math build omits the H3 CUDA sources to keep bitwise parity, so H3 CUDA tests skip there)

Scope: block-0 h3_block_replay --layouts tiny,small,medium, the full 50-block model qualification (below), and a one-step generation preflight. The three-seed generation gates (about 20 h per seed) were not run on this machine.

Figures

Full 50-block model on H100 (one packed FL2VA sample, forward plus complete backward, 638 parameter gradients; files under pr539/h100-full-model/ on the pr_assets branch, commit 23cc7e28)

All references and bounds were re-measured on H100: FP32 reference plus four BF16 providers (default, sdpa-flash, sdpa-efficient, cuBLASLt).

Check	Result
Block-0 golden gates	9 passed, 0 skipped
Ensemble rule calibrated on H100	k = 4.0 (the B200 rule uses 2.6); leave-one-out worst ratios 0.99 / 0.99 / 2.77 / 3.94
Candidate vs ensemble rule	passes, 0 failures, worst ratio 0.577
Candidate vs the single default-provider bound (1.5x provider error + floor)	fails; 347 tensors exceed it, first drift at nodes.block.33
Output error vs FP32	video 0.0063, audio 0.0198 (default provider 0.0064 / 0.0214)
Generation preflight (128x128, 1 step)	completed; flow check only, not the three-seed evidence

The single-provider bound is not the acceptance criterion here, for these reasons:

• Outputs and input gradients match or beat the default provider. Only the parameter gradients of the attention weights in the 50 main blocks are higher (about 1.65x; relative L2 around 0.5 vs 0.3).
• The baseline is set in the backward of the last block: the gradient error is 0.021 for both candidate and provider at block 49, and jumps to 0.39 (provider) / 0.78 (candidate) at block 48, then keeps that ratio in every earlier block.
• Block 49's attention is ill-conditioned (maximum logit about 3189, mean softmax entropy 0.57; other blocks 14 to 536 and 1.0 to 1.5). A bf16-sized perturbation of its inputs moves dk by 0.16 (median) to 0.65 (95th percentile); other blocks move by 0.003 to 0.005.
• With identical inputs the candidate attention is more accurate than SDPA at block 49 (out/dq/dk/dv error 0.0014/0.0010/0.0016/0.0016 vs flash 0.0015/0.0092/0.0043/0.0018). The other BF16 providers spread widely at the same place (block-48 gradient error: cuBLASLt 0.39, efficient 0.45, flash 0.68); the candidate is at the high end of that spread, inside the ensemble bound.
• Keeping q/k in fp32 only inside block 49 was evaluated and is not expected to help (the error in q/k is inherited from earlier blocks), so no kernel or numerical-contract change was made.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 221bb41f-2665-4833-997a-10f4a0946518

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[fusheng-ji]

Superseded by a stack of three PRs so that each layer is at most 10,000 lines of its own diff: #586 (block 0 replay and diagnostics), #587 (full-model qualification) and #588 (generation gate).

The top layer has the same tree as this PR's head 17d0e9e, except nine docs/operators/h3-*.md files whose commit-hash text was updated in #483–#494 after this PR was cut (17 lines).

Closing; please review the new PRs. The branch of this PR (feat/h3-one-block) is kept because #567–#575 are stacked on it.