[fusheng-ji]

RFC #420 stack 10a/10 (base test-h3, merge bottom-up): #483 → #484 → #485 → #486 → #487 → #488 → #489 → #493 → #494 → this PR (10a) → 10b (#587) → 10c (#588). This PR merges after #494; until the lower PRs merge, GitHub also shows their commits. This PR's own diff.

Replaces #539 (closed, superseded by this stack). #539 (28k lines on GitHub, 13.8k of its own over #494) is split into three layers so that each is at most 10,000 lines of its own diff: this PR (10a, block 0 replay, 3.6k), 10b (full-model qualification, 8.8k), 10c (generation gate, 1.5k). The top layer has the same tree as #539's head 17d0e9e9 apart from nine docs/operators/h3-*.md files whose commit-hash text was updated in #483–#494 after #539 was cut (17 lines).

Evidence. Measurement outputs (logs, JSON, figures) are not part of this diff. They are on the fork's pr_assets branch and every link is fixed to commit 967000e9. Runs on the heads of this stack: B200 pending/b200-split-539, H100 pending/h100-split-539 (each with tests.json, redaction.json and SHA256SUMS). Evidence of the earlier #539 runs: pr539/.

This stack

layer	PR	own diff	content
10a	#586	3.6k	block 0 replay and diagnostics
10b	#587	8.8k	full-model qualification
10c	#588	1.5k	generation gate

What

RFC #420 row ws1_one_h3_block: one real H3 block, forward and backward, with every intermediate and a first-drift report.

• The block. Block 0 of MiniMax-H3@42ed227 runs as a 17-node graph, from the AdaLN projection to the second gated residual. It runs on packed FL2VA layouts whose position_ids and tags are bitwise equal to the pinned diffusers pipeline's build_packed_sequence; the hashes are pinned in a CPU test.
• Three executions per node. Each node runs as the diffusers replay (op for op, no diffusers import), as the RL-Kernel binding (chained, and isolated, fed diffusers' inputs), and as an FP32 golden.
• Ownership. Each node names the RFC row that owns its arithmetic, and whether it is bound to that row's operator, an interim deterministic operator, or the provider replay.
• Forward report, per node: repeat and batch-row bitwise checks; chained and isolated agreement with diffusers (including the first differing packed row); candidate and diffusers error against the golden; the node's sha256; first_{,isolated_,repeat_,batch_}drift.
• Backward report: gradients of the block input, temb and all 12 block parameters (repeat, error vs golden); the gradient reaching every node (repeat, batch row), with first_grad_{repeat,batch}_drift.
• Weights. Block-0 attention/FFN weights are pinned in a new manifest section (block_tensors) and extracted by prepare_h3_weights.py --block from the same shard, so the conditioning file is unchanged.
• Tools. h3_block_replay.py, h3_block_ulp_sweep.py, h3_block_diagnostic.py, h3_first_step_diagnostic.py (replays a first denoising step from saved inputs; its results are in 10c), plot_h3_block.py, diagnose-block.sbatch, docs/operators/h3-one-block.md.
• Carried from test-h3. About 1.1k lines (deterministic_attention.cu, h3_full_attention.py, their test and benchmark, docs/operators/attention.md) come from [WS1][CUDA][MiniMax-H3] Add batch-invariant full attention #553, which test-h3 already has and [WS2][CUDA][MiniMax-H3] sp_norm_adaln: sequence-parallel norm/AdaLN/gated residual, byte-equal to WS1 (RFC #420 stack 9/10) #494's branch predates. They disappear from this diff once the stack is rebased on test-h3.

Bindings

Node	Owning row	Bound to
adaln_projection	adaln_projection_3mod	row (#485), via the registry
norm1, norm2	h3_rmsnorm	row (#487), via the registry
residual_attn, residual_mlp	adaln_gate_residual	row (#488), via the registry
q/k/v_proj, o_proj, ffn_gate_up, ffn_down	h3_qkv_gemm, h3_attention_o_gemm, h3_ffn_gate_up_gemm, h3_ffn_down_gemm	interim: Triton FP32-accumulate GEMM, one BF16 rounding
q_norm, k_norm	h3_qk_rmsnorm_d128	interim: #487's plain H3RMSNormCudaOp on the 128-wide heads (bitwise equal to nn.RMSNorm)
rope_q, rope_k	h3_mm_rope_3axis_partial	reference: the provider replay, since no MM-RoPE kernel exists yet
attention	h3_full_attention	interim: DeterministicAttentionOp(causal=False)
swiglu	h3_swiglu	interim: SwiGLUCudaOp

When an owning row lands, swapping its node's candidate callable is the whole integration, and the report entries become that operator's block-level evidence.

Prior art & reuse decision

This is a validation closeout, not a new kernel; no operator is written here.

• diffusers. MiniMaxH3TransformerBlock is the executable reference. It is replayed op for op as the provider (h3_provider), which pins its dtypes and op order without importing diffusers.
• [ws1]: WS1 Full Qwen3-8B Dense Train-Inference Closeout #315 Qwen3-8B chain gate (rl_engine/validation/models/chain_gate.py). Its method is reused: per-node digests and first drift in graph order. Its graph is Qwen3's (causal, no AdaLN, Qwen shapes), so the H3 block is built on the h3_chain stage pattern of [WS1][CUDA][MiniMax-H3] timestep_sinusoid_h3: FP32 timestep features + shared H3 harness (RFC #420 stack 1/10) #483.
• Interim operators are reused from RL-Kernel. The one decision taken here is the GEMM entry point. TritonDetGemmOp.__call__ reduces K as a tree with BF16 nodes; its FP32-accumulate kernel is equally deterministic and row-invariant, matches cuBLAS's accuracy, and is 9–17× faster (B200, error vs FP64):

Shape	Tree path	FP32 K loop → BF16	cuBLAS	Tree time	FP32 K loop time
264 × 5376 → 7168	7.0e-3	2.4e-3	2.4e-3	10.4 ms	1.2 ms
925 × 14336 → 5376	7.5e-3	2.6e-3	2.6e-3	47 ms	2.7 ms

The reuse candidates of the rows below are compared byte for byte by tools/validation/models/h3_bitwise_reuse.py (committed in #483).

Results

B200

Block-0 replay, generated from a clean tree at 41e4293 on an otherwise idle B200 (torch 2.13.0+cu130), with tools/validation/models/h3_block_replay.py --layouts tiny,small,medium: report.json.

S	First drift (chained / isolated)	Repeat or batch drift, any node or gradient	Output err: RL-Kernel / diffusers	Fwd ms	Fwd+bwd ms
264	adaln_projection / adaln_projection	none	5.21e-3 / 5.21e-3	7.4 vs 0.61	22.7 vs 2.8
925	adaln_projection / adaln_projection	none	7.87e-3 / 7.87e-3	30.1 vs 1.12	81.7 vs 5.0
3160	adaln_projection / adaln_projection	none	8.97e-3 / 8.97e-3	225 vs 3.8	577 vs 11.5

• Isolated bitwise promises. Every node that promises it is bitwise equal to diffusers on diffusers' inputs at every size: norm1/norm2, q_norm/k_norm, RoPE, and both gated residuals. The first drift is the first reduction whose tree differs from cuBLAS.
• Forward accuracy. Per-node error vs the FP32 golden is 0.93–1.20× diffusers'. The largest is the interim attention at S = 3160.
• Gradient accuracy. The candidate is up to 7× more accurate than diffusers on adaln_proj.weight, adaln_proj.bias and temb (diffusers' index_select backward accumulates in BF16 atomics; the row operators use FP32 segment sums). It is within 1.16× on the other leaves, except attn.norm_q.weight at S = 3160 (9.5e-3 vs 5.5e-3); at S = 925 the same leaf is 2× more accurate, so the 128-element gradient's error follows BF16 rounding, not one side.
• Control H-C3, reported, not promised. Permuting the packed rows with their positions and indices changes 79 / 450 / 2199 output rows (max 8). Attention sums keys in packed order; settling this belongs to h3_full_attention.
• Speed. The 12–60× forward cost is the interim GEMMs and attention (FP32 scores materialised), not the row operators.

H100 (Hopper)

Block-0 replay on 4 × NVIDIA H100 (sm90), driver 580, CUDA 13.0, torch 2.13.0+cu130, same pinned weights: figure and files on pr_assets.

Status

• Block-0 replay has no known failures (evidence at 41e4293 / caaf754, Results).
• Block-0 gradient bound. The block-0 backward test compared each gradient's relative error with 1.5 × diffusers' + 5e-3. For a BF16 leaf that number is set by one rounded element and moves in whole ulps; it failed on one seed with torch 2.13 and another with torch 2.9.1 although no row operator moves it. A BF16 leaf now also passes at most 6 ulps above diffusers'.
• Fast-math guard (from the #483 review). KERNEL_ALIGN_USE_FAST_MATH=1 adds --use_fast_math to the whole _C build, which approximates expf/sinf/cosf/rsqrtf/division and flushes denormals. All H3 bitwise kernels are now gated on one guard, so a fast-math build omits them and the dispatcher falls back to the PyTorch reference.
• Fixed: flaky test_h3_adaln_modulation.py::TestFusedModulation::test_backward_keeps_table_gradient_fp32 in [WS1][CUDA][MiniMax-H3] adaln_row_gather: fused six-way AdaLN row gather (RFC #420 stack 4/10) #486 (62f8009e), merged here. Its golden rounded an FP64 SiLU where the kernel rounds the FP32 SiLU, and temb was unseeded; the kernel was deterministic.
• DCO. The DCO check can report a failure on this stack because it cannot read more than 250 commits; every commit authored for this layer carries Signed-off-by.
• Full-model qualification is in 10b; the generation gate in 10c.

Tests

Layer head 193f079e. Pinned MiniMax-H3@42ed227 weights including block 0.

python tools/weights/prepare_h3_weights.py --out <dir> --block && export RL_KERNEL_H3_WEIGHTS=<dir>
RL_KERNEL_H3_STRICT=1 python setup.py build_ext --inplace
python -m pytest tests/models/minimax_h3 tests/backends/cuda/test_h3_full_attention.py -q
# 4/8-GPU NCCL byte-equality (do not run under torchrun; WORLD_SIZE must be unset)
python -m pytest tests/models/minimax_h3/test_h3_sp_norm_adaln.py tests/models/minimax_h3/test_h3_tp_adaln.py -k nccl -v

Check	B200	H100
tests/models/minimax_h3 + test_h3_full_attention.py (2 GPUs)	540 passed, 4 skipped (sglang and matplotlib on the Python path)	542 passed, 2 skipped
test_h3_timestep_sinusoid.py + test_h3_report.py with sglang 0.5.22 and matplotlib	49 passed (the sglang parity test and the plots run)	included in the line above
NCCL byte-equality, sp_norm_adaln + tp_adaln	6 passed (2 / 4 / 8 GPUs) on one 8 × B200 node, measured on the heads before the re-cut (cbdf48b0, 44b7f832; same layer code, nine docs files differ)	4 passed (2 / 4 GPUs); the 8-GPU cases were not run (nodes have 4 GPUs)
pre-commit, mypy (--ignore-missing-imports rl_engine/)	clean

The 4 B200 skips are the 4- and 8-GPU NCCL cases (2 visible GPUs); they pass in the 8-GPU run above (pre-re-cut heads). Without sglang and matplotlib on the path, two more tests skip (538 passed, 6 skipped). 8-GPU NCCL is validated on B200 only.

tests/models/minimax_h3/test_h3_one_block.py checks the block-0 replay: on CPU, that the layouts are pinned to diffusers, the graph is well formed and MM-RoPE rotates 96 of 128 channels; on GPU forward (S = 264, 925), that every node repeats and is batch-row bitwise, promised nodes are isolated-bitwise and node and output errors are at most 1.25 × diffusers' + 1e-3; on GPU backward (S = 264), the gradient checks described in Status (every leaf repeats; the ulp-based bound for BF16 leaves).

Fast-math regression tests: tests/build/test_build_platform_collectives.py::test_cuda_build_compiles_h3_kernels_without_fast_math, ::test_fast_math_build_excludes_bitwise_h3_kernels, tests/runtime/test_dispatch.py::test_h3_timestep_falls_back_when_fast_math_build_omits_kernel, ::test_h3_ops_prefer_cuda_and_fall_back_to_pytorch_reference.

On this PR the GitHub checks are DCO (see Status), CodeRabbit, lint, the CPU unit tests and docs.

Scope / Known limitations

• ROCm. CUDA only. The interim attention and Triton GEMM build for ROCm, but the block has not been run on AMD hardware.
• Interim bindings. 11 of 17 nodes are interim or reference. Their speed and rounding are not their owning rows' final ones, and RoPE carries no RL-Kernel evidence until h3_mm_rope_3axis_partial lands.
• Stand-in inputs. The block input is a seeded stand-in (the input projections are other rows), and temb is the provider's (the conditioning chain is [WS1][CUDA][MiniMax-H3] timestep_sinusoid_h3: FP32 timestep features + shared H3 harness (RFC #420 stack 1/10) #483–[WS1][CUDA][MiniMax-H3] adaln_row_gather: fused six-way AdaLN row gather (RFC #420 stack 4/10) #486).
• Block 0 only. The 17-node replay runs block 0; the 50-block and generation gates are 10b and 10c.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}