[fusheng-ji]

RFC #420 stack 10b/10 (base test-h3, merge bottom-up): … → #494 → 10a (#586) → this PR (10b) → 10c (#588). This PR merges after 10a. This PR's own diff (8.8k lines, 6.9k of them three JSON data files).

Replaces #539 (closed, superseded by this stack). The split is described in 10a. This layer holds the full-model qualification; the generation gate is 10c.

Evidence. Measurement outputs (logs, JSON, figures) are not part of this diff. They are on the fork's pr_assets branch and every link is fixed to commit 967000e9. Runs on the heads of this stack: B200 pending/b200-split-539, H100 pending/h100-split-539 (each with tests.json, redaction.json and SHA256SUMS). Evidence of the earlier #539 runs: pr539/.

This stack

layer	PR	own diff	content
10a	#586	3.6k	block 0 replay and diagnostics
10b	#587	8.8k	full-model qualification
10c	#588	1.5k	generation gate

What

RFC #420 closeout row ws1_full_h3_transformer, numerical part: the real 50 transformer blocks, both text-refiner layers and the input/conditioning/output components, forward and backward, against a frozen reference.

• tools.validation.models.h3_full_model runs the full model. Each candidate block binds its own parameters to the block-0 graph of 10a, and the row/interim/reference labels are kept. Every parameter and input gradient is checked.
• A reference pass (independent FP32 and BF16 provider runs) freezes per-tensor error bounds with a committed formula: 764 tensor bounds in examples/models/minimax_h3/full-model-thresholds.json, a provider-ensemble rule in full-model-ensemble-rule.json, and a 1,635-file SHA-256 reference inventory (reference-manifest.json).
• The candidate gate (candidate.sbatch, h3_candidate_prepare.py, h3_full_compare.py) builds its own H3 extension, verifies every reference and checkpoint hash, then runs the block-0 golden tests (no skips), full-model forward/backward with every candidate node required to run (CUDA fallback rejected) and first drift ordered by real forward/backward execution. Only a passing numerical comparison lets the generation gate (10c) start.
• h3_full_provider_spread.py and h3_full_ensemble_rule.py produce the provider spread and the acceptance rule; --provider-variant and --provider-nodes on the full-model runner swap in provider configurations or chosen provider nodes.
• examples/models/minimax_h3/README.md documents the kit; reference.sbatch regenerates the reference. The scheduler scripts keep only site-neutral resource requests.

Prior art & reuse decision

No operator is written here. The reference is the diffusers pipeline itself (FP32 and BF16), run in four distinct provider configurations (default, flash attention, memory-efficient attention, cuBLASLt) so that the acceptance rule is calibrated on provider-to-provider spread, not on a single run of one configuration.

Results

B200

Gates of candidate.sbatch at c249a735 on one B200 (torch 2.9.1+cu130) against the regenerated reference, from a clean checkout. Evidence: README, acceptance.

Gate	Result
Block-0 golden tests	9 passed
Full-model forward/backward (50 blocks, both refiners, boundaries), 764 tensors, provider-ensemble rule	passed, 0 failing; worst tensor at 0.55 of its bound (block 39 adaln_proj.linear.weight)
Final outputs vs the FP32 reference, relative_l2	video 0.0063 (0.35 of its bound), audio 0.0218 (0.39)
The original single-configuration bounds (kept as a record)	54 of 764 fail: 52 on max_abs only, 2 also on relative_l2

H100 (Hopper)

One packed FL2VA sample, forward plus complete backward, 638 parameter gradients, on 4 × NVIDIA H100 (sm90), torch 2.13.0+cu130; files under pr539/h100-full-model/ on pr_assets (commit 23cc7e28). All references and bounds were re-measured on H100: FP32 reference plus four BF16 providers.

Check	Result
Block-0 golden gates	9 passed, 0 skipped
Ensemble rule calibrated on H100	k = 4.0 (the B200 rule uses 2.6); leave-one-out worst ratios 0.99 / 0.99 / 2.77 / 3.94
Candidate vs ensemble rule	passes, 0 failures, worst ratio 0.577
Candidate vs the single default-provider bound (1.5x provider error + floor)	fails; 347 tensors exceed it, first drift at nodes.block.33
Output error vs FP32	video 0.0063, audio 0.0198 (default provider 0.0064 / 0.0214)

The single-provider bound is not the acceptance criterion, for these reasons:

• Outputs and input gradients match or beat the default provider. Only the parameter gradients of the attention weights in the 50 main blocks are higher (about 1.65x; relative L2 around 0.5 vs 0.3).
• The baseline is set in the backward of the last block: the gradient error is 0.021 for both candidate and provider at block 49, and jumps to 0.39 (provider) / 0.78 (candidate) at block 48, then keeps that ratio in every earlier block.
• Block 49's attention is ill-conditioned (maximum logit about 3189, mean softmax entropy 0.57; other blocks 14 to 536 and 1.0 to 1.5). A bf16-sized perturbation of its inputs moves dk by 0.16 (median) to 0.65 (95th percentile); other blocks move by 0.003 to 0.005.
• With identical inputs the candidate attention is more accurate than SDPA at block 49 (out/dq/dk/dv error 0.0014/0.0010/0.0016/0.0016 vs flash 0.0015/0.0092/0.0043/0.0018). The other BF16 providers spread widely at the same place (block-48 gradient error: cuBLASLt 0.39, efficient 0.45, flash 0.68); the candidate is at the high end of that spread, inside the ensemble bound.
• Keeping q/k in fp32 only inside block 49 was evaluated and is not expected to help (the error in q/k is inherited from earlier blocks), so no kernel or numerical-contract change was made.

Status

• Numerical acceptance of the full model: passed (B200 gates at c249a735; H100 re-measured as above).
• How the acceptance rule came about.
  • The bounds frozen from the reference run (1.5 × one provider pass + floor) failed 54 of 764 tensors for the candidate. The same bounds also fail diffusers runs that differ only in attention backend or BLAS library: flash attention 36 tensors on max_abs (1 on relative_l2), cuBLASLt 38 (3), memory-efficient attention 66 (9), while the default configuration they were calibrated on fails none.
  • A diagnostic that ran the full model with chosen block nodes replaced by their diffusers replay (attention, the six GEMMs, SwiGLU, Q/K norm, the two norms, the residuals, the AdaLN projection, all interim operators, all nodes) found no single group responsible.
  • The acceptance rule is therefore calibrated on four distinct provider configurations only, with no candidate data: relative_l2 ≤ k × the largest provider error + floor, with k = 2.6 the smallest factor at which every provider configuration, held out in turn, passes against the other three. max_abs, which is set by one element, is reported but not gated.
  • The rule was designed after the candidate had failed the earlier bounds. That is why the comparison against the old bounds is kept in the evidence next to the new decision.
• The reference was regenerated. The first reference run's artifacts (about 200 GB) were deleted to save disk space, so they were regenerated with the original commands at the same source commit (2681cadb). 1,614 of the 1,635 pinned files are byte-identical to the first run's; the others are the eight report.json files, the thresholds file that embeds their hashes and twelve video.mp4 / audio.wav containers, which also differ between two runs of one job.
• The 764 bounds and the ensemble rule are data files read by the gate (candidate.sbatch, h3_full_compare.py), so they stay in this diff (6.9k of its 8.8k lines).
• The generation quality gate is not passed; see 10c.

Tests

Layer head 5ecb92de; same commands as 10a.

RL_KERNEL_H3_STRICT=1 python setup.py build_ext --inplace
python -m pytest tests/models/minimax_h3 tests/backends/cuda/test_h3_full_attention.py -q

Check	B200	H100
tests/models/minimax_h3 + test_h3_full_attention.py (2 GPUs)	549 passed, 4 skipped (sglang and matplotlib on the path)	551 passed, 2 skipped
full tests/ (minus tests/distributed and a ROCm import-error file)	73 failed, 3519 passed; the same 73 fail on upstream/test-h3 (measured on the heads before the re-cut; same layer code)
pre-commit, mypy	clean

Skips: as in 10a (the 4- and 8-GPU NCCL cases on B200; the 8-GPU cases on H100). The 73 failures are on test-h3 as well; none is introduced by this layer.

tests/models/minimax_h3/test_h3_full_model.py is the unit test of the full-model runner added in this layer.

Scope / Known limitations

• Boundary components (input projections, conditioning, output heads) stay provider/reference.
• Acceptance rules. The full-model numerical gate is relative_l2 only (max_abs is reported); it rests on measured provider spread and was designed after the candidate had failed the earlier bounds.
• Reference artifacts. The first reference run's artifacts were deleted and regenerated; the pinned inventory is re-pinned to the regenerated files, 21 of 1,635 of which differ from the first run's (metadata and media containers).
• CUDA only; not run on AMD hardware.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}