GPU RMSNorm runs with ε = 1e-5, not the model's 1e-6: the kernel cache key omits ε and preload bakes 1e-5 — special-token positions come out 0.431× (qwen2.5-coder-7b, F2 pos 13/44) #3759

qwen2.5-coder-7b Q4_K_M, 0.69.1 row (#3710). After #3728 the pos-1 divergence is gone, but the F2 gate still rejects the GPU at pos 44 (0.69.0's double chat template, cosine ≈0.935) or pos 13 (the #3672 clean prompt, cosine 0.883), with argmax equal. It fails on lambda on both prefill paths, and on gx10 only with BATCHED_PREFILL=1.

It is the GPU, and it is the RMSNorm epsilon

• Ground truth (float64: layer-0 Q/K/V weights dequantized from the GGUF + bias, applied to the dumped attn_norm input, pos 0 where RoPE is the identity): cos(truth, CPU) = 1.000000 for Q, K and V. cos(truth, GPU) = 0.994 (Q), 0.99993 (K, masked by its bias norm of 668), 0.738 (V). The CPU reference is exact; the GPU is wrong.
• The GPU's W·x has the right direction (cos 1.0000) and the wrong size: 0.431× the truth, for both Q and V. So its input is scaled.
• The per-position norm ratio GPU/CPU of the attn_norm output (apr parity --per-op, serial path, prompt <|im_start|>user\nWhat is the capital of France? Answer briefly.<|im_end|>\n<|im_start|>assistant\n, lambda) fits RMSNorm with ε = 1.00e-05 at all 17 positions, against the model's rms_norm_eps = 1e-6. Ordinary tokens (embedding mean-square ≈3e-4) come out 0.986×. <|im_start|> / <|im_end|> have near-zero embeddings (mean-square 1.05e-6) and come out 0.431×, which corrupts Q and V exactly at the special-token positions. The cosine check on attn_norm read 1.000 because cosine is scale-invariant.

Mechanism

ε is a PTX immediate (VectorizedRmsNormKernel / BatchedVectorizedRmsNorm: ctx.mov_f32_imm(epsilon)), but the module cache keys omit it: rmsnorm_vectorized_{hidden} and batched_rmsnorm_vectorized_{hidden} (cuda/executor/layer_norm_gpu.rs). preload_rmsnorm_module and preload_batched_prefill_modules (cuda/executor/layers/modules_utilities.rs) compile those keys with a hardcoded epsilon: 1e-5. Every later launch that passes the model's 1e-6 finds the cached 1e-5 kernel. At cc ≥ 120 the serial path uses rmsnorm_simple_{hidden}, which is never preloaded and so compiles with the real ε. That is why gx10's serial default is green and its batched path is not.

Blast radius: every model whose rms_norm_eps ≠ 1e-5 (Qwen2.5 and Qwen3 are 1e-6), on every chat prompt, because the special tokens are in every template.

done_when

• Every RMSNorm module key includes ε (its bits), and preload compiles with the model's ε, so a cached kernel can never carry a different ε than the launch asked for.
• A GPU test with a near-zero-mean-square input (≈1e-6) and ε = 1e-6: the GPU RMSNorm output norm matches the f32 reference within 1e-3 relative. With a 1e-5 kernel preloaded under the old key it goes RED.
• qwen2.5-coder-7b Q4_K_M: GPU backend with no fallback, apr run correct, apr qa golden 3/3, on lambda (default) AND on gx10 with BATCHED_PREFILL=1. The attn_norm GPU/CPU norm ratio is 1.000 at the special-token positions.

Refs #3710 #3728 #3751

Acceptance widened by the cop (aprender-04, 2026-09-21), verbatim

"Two widenings before you code: 1. Qwen3.5 is probably hit too. […] Put a qwen35 cell (9B, chat prompt, lambda + gx10) in the acceptance. 2. Siblings. This is the SECOND cache-key-scope defect today (#3727: the FP8 activation cache key). Sweep every PTX/JIT module cache: for each key, list every immediate baked into the PTX (eps, rope theta, head dims, scales…) and show it's in the key or the kernel reads it at runtime. Add a guard/test that fails when a module's key omits a baked parameter. […] Acceptance as you wrote it, plus those two."

• done_when 4 (Qwen3.5): a Qwen3.5-9B chat-prompt cell on lambda and gx10, 0.69.0 against the fix.
• done_when 5 (siblings): every PTX/JIT module-cache key either carries each parameter its kernel bakes in (epsilon, RoPE theta, head dims, scales) or reads it at runtime, shown per key. A guard/test fails when a key omits a baked parameter. Any sibling key the sweep finds incomplete is fixed in this change.