[fusheng-ji]

RFC #420 stack 10c/10 (base test-h3, merge bottom-up): … → #494 → 10a (#586) → 10b (#587) → this PR (10c). This PR merges after 10b. This PR's own diff (1.5k lines). The H3 PRs #567 and #568 → #575 are stacked on the top of this layer.

Replaces #539 (closed, superseded by this stack). The split is described in 10a. The top of this stack has the same tree as #539's head 17d0e9e9 apart from nine docs/operators/h3-*.md files whose commit-hash text was updated in #483–#494 after #539 was cut (17 lines).

Evidence. Measurement outputs (logs, JSON, figures) are not part of this diff. They are on the fork's pr_assets branch and every link is fixed to commit 967000e9. Runs on the heads of this stack: B200 pending/b200-split-539, H100 pending/h100-split-539 (each with tests.json, redaction.json and SHA256SUMS). Evidence of the earlier #539 runs: pr539/.

This stack

layer	PR	own diff	content
10a	#586	3.6k	block 0 replay and diagnostics
10b	#587	8.8k	full-model qualification
10c	#588	1.5k	generation gate

What

RFC #420 closeout row ws1_dual_trajectory, generation part: three-seed generation with the candidate and its comparison against the provider, built to survive preemption. Not claimed as passed (see Status).

• Generation follows examples/models/minimax_h3/generation-contract.json: the pinned public reference image and prompt, seeds 42/43/44, 124 frames, 1344×768, 24 fps with joint 32 kHz audio. generation-compare.sbatch and h3_generation_compare.py check identical initial noise and shapes and report PSNR, SSIM, temporal RMSE and audio differences against the provider.
• Resumable generation (h3_full_generation.py, rl_engine/validation/models/h3_generation_resume.py). Candidate generation takes about 20 hours per seed, so it runs as one resumable task per seed. Each saved step has a sidecar with the digest of the transformer's inputs and of the saved file, written atomically. On resume the saved transformer calls are replayed from disk and the pipeline's own scheduler turns them into the next latents; every replayed call's input digest is checked. Recomputing the last saved step and requiring a bitwise match is --verify-last-step (opt-in).
• A run is tied to the git trees of the model paths, not to a commit, so a commit that changes only orchestration code can continue it; h3_generation_adopt.py moves an older commit-based run over, only if those paths are byte-identical.
• Job scripts: generation.sbatch runs one seed per array task and refuses to start unless the numerical gates of 10b passed; generation-resume-check.sbatch checks resume; generation-compare.sbatch compares afterwards.
• --provider-variant on h3_full_generation generates the provider's videos with one SDPA backend forced on the transformer's forward only, for the quality-bounds comparison below.

Prior art & reuse decision

No operator is written here. The reference is the diffusers pipeline; its own generation is the baseline, including variants with one SDPA backend forced.

Results

B200

Generation from clean checkouts at 4792c8a9 / 2451ae5c (same model code), 124 frames 1344×768, 49 steps. Evidence on pr_assets: h3-full-model-b200/.

Gate	Result
Generation preflight (128×128, one step)	complete
Three-seed generation, 124 frames 1344×768, 49 steps	complete
Generation quality vs the provider, bounds fixed before generation	not passed: seed 44 passes, seeds 42 and 43 do not

Generation quality per seed (bounds: PSNR ≥ 30 dB, SSIM ≥ 0.99, temporal RMSE ≤ 0.01, audio relative L2 ≤ 0.05; a failing value is bold):

seed	PSNR (dB)	SSIM	temporal RMSE	audio relative L2	passed
42	37.58	0.9917	0.00991	0.0807	no
43	32.64	0.9757	0.01056	0.0762	no
44	38.75	0.9945	0.00734	0.0456	yes

The same bounds applied to the provider with one SDPA backend forced on the transformer (same seeds, inputs and conditioning):

provider variant	seeds passing	PSNR (dB), seeds 42 / 43 / 44	audio relative L2, seeds 42 / 43 / 44
flash attention	0 of 3	39.08 / 23.98 / 39.33	0.054 / 0.264 / 0.056
memory-efficient attention	0 of 3	39.43 / 30.22 / 42.59	0.094 / 0.135 / 0.116
candidate	1 of 3	37.58 / 32.64 / 38.75	0.081 / 0.076 / 0.046

First denoising step replayed from its saved packed inputs (nothing has fed back yet), relative L2 of the video output against the provider's saved step (replaying the provider is bitwise equal to it):

configuration (seed 43 unless noted)	video relative L2
candidate (seeds 42 / 43 / 44)	0.0285 / 0.0212 / 0.0274
provider, flash / memory-efficient attention	0.0219 / 0.0218
provider, cuBLASLt	0 (bitwise equal at this step)
candidate with attention / the six GEMMs / SwiGLU / the AdaLN projection run as the provider	0.0225 / 0.0217 / 0.0198 / 0.0193
candidate with Q/K norm, the two norms or the residuals run as the provider	0.0212 (unchanged by the swap)
candidate with all interim operators / every node run as the provider	0.0192 / 0

H100 (Hopper)

A one-step generation preflight (128×128) completed on 4 × NVIDIA H100; it checks the flow only and is not quality evidence. The three-seed generation (about 20 hours per seed) was not run on H100.

Status

• Generation quality: not passed. Against the provider, with the bounds fixed before candidate generation, seed 44 passes and seeds 42 and 43 do not: audio relative L2 is 0.081 and 0.076 (bound 0.05), and seed 43 also misses SSIM (0.9757) and temporal RMSE (0.01056). The bounds are unchanged.
• What the comparison says about those bounds.
  • The bounds had only been checked against a bitwise-identical repeat of the provider (PSNR 300 dB, zero error). The provider with one SDPA backend forced on the transformer, run through the same 49 steps with the same seeds, inputs and conditioning, fails them too: flash attention 0 of 3 seeds, memory-efficient attention 0 of 3, and audio relative L2 exceeds 0.05 on every seed of both (0.054 to 0.264).
  • The first denoising step shows the same: the candidate differs from the provider's step by 0.021 to 0.029 relative L2 on video, and the provider with flash or memory-efficient attention differs from its own default by 0.022. Running any one group of block nodes as the provider's leaves 0.019 to 0.023; replacing the norms or the residuals changes nothing at this step.
  • This is not a claim that the candidate is better than the provider variants: on seeds 42 and 44 its PSNR and temporal RMSE are slightly worse than both variants', on seed 43 better, and it passes one seed where they pass none.
  • The generation bounds were not changed and the candidate is not accepted on generation quality. A rule that tolerates a change of kernel, like the one for the numerical gates in 10b, would be a second rule designed after seeing a failure; it is left to the reviewers.
• Resume checks. On the real pipeline at preflight size (128×128, 6 sampling steps), one uninterrupted run and runs preempted and resumed give byte-identical step files, initial inputs, video and audio (8 files); a resume with nothing left to compute and the opt-in verification both reproduce the uninterrupted run.
• Cost. Candidate generation is more than 50× slower than the provider's (about 20 hours per seed, against about 1 h for the provider's).

Tests

Layer head 6fe346df; same commands as 10a.

RL_KERNEL_H3_STRICT=1 python setup.py build_ext --inplace
python -m pytest tests/models/minimax_h3 tests/backends/cuda/test_h3_full_attention.py -q

Check	B200	H100
tests/models/minimax_h3 + test_h3_full_attention.py (2 GPUs)	571 passed, 4 skipped (sglang and matplotlib on the path)	573 passed, 2 skipped
full tests/ (minus tests/distributed and a ROCm import-error file)	73 failed, 3541 passed; the same 73 fail on upstream/test-h3 (measured on the heads before the re-cut; same layer code)
pre-commit, mypy	clean

Skips as in 10a. tests/models/minimax_h3/test_h3_generation_resume.py (22 CPU tests, included in the counts) drives a fake denoising loop through the same resume code: a run preempted once or repeatedly equals an uninterrupted one, nothing is recomputed by default, the opt-in verification catches a drifting recompute, other identities, corrupted files and signature changes are rejected, and adopting a commit-based run is tested.

Scope / Known limitations

• Generation quality is not passed (1 of 3 seeds), against bounds that the provider with another attention kernel does not pass either.
• The three-seed generation was run on B200 only.
• CUDA only; not run on AMD hardware.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}