CUDA kernel contract on nvcc 13.0: 656 pending launches have no 13.0 records, so kernel-contract and kernel-contract-limits fail on plain v1.1.1 #1105

What happens

On plain v1.1.1 (b10231b9), with nvcc 13.0 on a GB10 (sm_121), the CUDA kernel contract passes every gate and fails only on records:

• tf-cuda-test kernel-contract: SUMMARY 85 kernels: 85 pass gates 1-4, 0 fail, 0 known findings; 7 controls: 7 red as planted, 0 missed; 1533 limit problems; …, then FAIL kernel-contract: 0 gate failures, 0 controls missed, 1533 limit problems. The 1,533 are FAIL limits: 1101 registered CUDA launches against sm_80/86/89/120/121, 877 problems (each a <launch> on sm_N: no compiled usage recorded line) and 656 FAIL limits live lines.
• tf-cuda-test kernel-contract-limits on the five SMs' fatbins: FAIL kernel-contract-limits: 877 problems.

Both print images built by nvcc 13.0, checked against nvcc 13.0's records.

What the problems are

All of them are missing records, not launches over a limit:

• In kernel-contract-limits' output, no launch is over a limit: apart from the planted controls' 10 RED lines, all 2,990 limits sm_N lines are PASS. 2,113 end in (record current), and the other 877 in (record stale), the measured usage differing from the record. These 877 launch and SM pairs are the same ones kernel-contract lists as no compiled usage recorded.
• The 877 belong to 656 launches: 55 on all five SMs, affine4 verify32 cluster of 8 on sm_120 and sm_121, and 600 on sm_121 alone, the other four SMs printing N/A … below_minimum for them. None of the 656 has a current record on any SM.
• kernel-contract's other 656 are its FAIL limits live lines on sm_121, one for each of the same 656 launches, each with (recorded 0, 0), for example:
  FAIL limits live qwen35 tf_q35_gdn_slots_1: 128 threads launched, the driver allows 128; 48 registers, 0 + 0 shared bytes (recorded 0, 0).
• These are pending launches, whose usage is zero (dsv41_pending_1.zig and _2.zig, cuda_lane_pending.zig, inc4_pending.zig, qwen_decode_pending.zig, glm_sampling_pending.zig and affine_verify_pending.zig); 7 more pending dsv41_x3gm launches are N/A on all five SMs and print neither a problem nor a RECORD line. pending_132.lookup returns a measured usage only for nvcc 13.2 and later, and only for the launches measurements/*_132.zig lists. On 13.0 they keep none, and registry_pending_test.zig says that is intended ("Pending 13.0 launch records never qualify without that compiler's resources").
• By the launch name's first word: dsv41_x3gm 128, dsv41_lin2 96, dsv41_linear 45, dsv41_router_gemv 40, dsv41_exl3_experts 32, dsv41_dense3 27, dsv41_attn 24 and 42 more, among them qwen35 12 (16 with the unprefixed tf_q35_* launches of its image), gdn_prep 6, act_rows 6, tree_round 5 and glm_glue 1.

docs/kernel-contract.md names nvcc 13.0 as the compiler of the release fatbins ("registry.cuda holds what nvcc 13.0 compiles"). With 13.0, then, the contract cannot pass until these launches have 13.0 records. I found no mention of this in CHANGELOG.md or docs/receipts/, and none of the receipts there runs the contract.

The 13.0 measurements

kernel-contract-limits prints a RECORD line in registry.zig's form for each registered launch it checks on at least one SM, with an entry for each SM it checks: all five for the 55 launches, sm_120 and sm_121 for affine4 verify32 cluster of 8, and sm_121 alone for the other 600 (it prints no entry for an SM it reports N/A, where registry.zig keeps all five). The plain v1.1.1 run's output, with these lines, is in #1050's receipt: single-v1.1.1/limits.log. For example:

RECORD act_rows tf_dconv_rows: .usage = .{ .{ .regs = 32, .static = 0, .bound = 256 }, .{ .regs = 37, .static = 0, .bound = 256 }, … }

If 13.0 records for the pending launches are wanted, I can open a pull request with them as a set. Otherwise, the docs could say that the contract on 13.0 is expected to fail on the pending launches.

To reproduce

On a GB10 with CUDA 13.0, on v1.1.1:

zig build fatbins kernel-contract-fatbins -Dnvcc=/usr/local/cuda-13.0/bin/nvcc -Dsm=80,86,89,120,121 -p zig-out-kc
tf-cuda-test kernel-contract-limits zig-out-kc/fatbin
tf-cuda-test kernel-contract

tf-cuda-test was built from the same tree. On #1050's head (80517fc3, 91 kernels in the contract), the counts are the same: 1,533 and 877.

Not run: the release archive's own tf-cuda-test, and nvcc 13.2 or later.

Written with Weave (My Symbiotic AI)