Repository navigation
cuda act_rows: Dory's scaled output gate and scaled SwiGLU, each row then L2-normed (Dory, #1233) - #1258
Closed
jschmied wants to merge 4 commits into
Closed
cuda act_rows: Dory's scaled output gate and scaled SwiGLU, each row then L2-normed (Dory, #1233)#1258jschmied wants to merge 4 commits into
jschmied wants to merge 4 commits into
Conversation
…d lerp residual Dory's residual stream lives on the unit sphere, so its rows need two kernels norms.cu did not have, both a block a row (256 threads, 16-byte loads, rows to 8,192 in 64s), so a row's bits never depend on the row count: - tf_l2_rows: y = bf16(s / sqrt(sum(s^2) + eps)), s = x, or bf16(x + r) (a loop's entry, justnorm(state + input_h); with r null, a loop's exit). - tf_slerp_rows: dory.py's slerp_norm, h = bf16(l2(a + |alpha scale| (b - a))) with a = bf16(l2(res)) and b = bf16(l2(out)), the lerp in fp32 steps. Norms.l2 and Norms.slerp launch them; no existing kernel or launch changes. Both are registered with the kernel contract. Their check compares every output with float64's value at the bf16 rounding points as a range over the fp32 run's spread (core/l2_ref.zig); three controls each drop one rounding point (the sum before the norm, the slerp's a, its b) and go red at gate 3. Gate 4's integer rows make an fp16 sum overflow to a zero output (85x and 51x the bound); a bf16 sum's scale cancels in the norm, so that margin is stated, not proven, as for the RMS tails. Launch records from nvcc 13.0's sm_80/86/89/120/121 images.
…2 norm and q's scale tf_qk_rows_l2 takes dory.py's order: rotate-half RoPE on the projection's bf16 rows (rounded to bf16), then each head's L2 norm with eps on the sum, then q times an fp32 scale vector ((sqk f)^2, formed on the host as torch forms it) after the normalized row's rounding; k is its normalized row, v as is, and k and v go into the caches as tf_qk_rows writes them. Shape.l2 and Shape.q_scale select it; every other call keeps its kernel. It is registered with the kernel contract: every output against float64's range at its rounding points (cosf's ulps on the fp32 angle pos * inv, the inverse's fp32 spread), at head dim 256, 16 query heads over 4, a third of the rows past position 199,800. Two controls each drop one rounding point (the rotated head, q's normalized head) and go red at gate 3. Gate 4's integer heads (no rotation) overflow an fp16 sum to a zero output. core/l2_ref.zig's intervals now keep infinite ends and widen by fp32's least subnormal step, so a planted infinity leaves only the outputs float64 makes non-finite.
…then L2-normed Two activation rows for Dory, a block a row, 16-byte loads, fp32 scale vectors formed on the host as torch forms them: - tf_gate_l2_rows: out = bf16(l2(bf16(o * sigmoid(g * sg)))) over all heads of a row (dory.py's output gate, then justnorm over num_heads * head_dim); - tf_swiglu_l2_rows: out = bf16(l2(bf16(silu(bf16(g * sug)) * bf16(u * suu)))) (its MLP). sigmoid and silu are torch's fp32 forms, 1 / (1 + expf(-x)) and x / (1 + expf(-x)) with IEEE division. ActRows.gateL2 and ActRows.swigluL2 launch them; no existing launch changes. Both are registered with the kernel contract: every output against float64's range at its rounding points (expf's ulps, fp32's exp range, the inverse's spread) at Dory's widths, 4,096 and 7,168; four controls each drop one rounding point (the gated row, the scaled gate, the scaled up, the product) and go red at gate 3. Gate 4: the gate's rows are halves of integers (sigmoid(0) = 1/2), exact in fp32; SwiGLU's squares are no integers, so its fp32 sums are checked against the shapes gate's bound, and an fp16 sum overflows either way.
This was referenced Oct 24, 2026
Owner
|
Landed on main in 6528fbe..81b476d: the whole Dory stack (#1256-#1261), with your records under your authorship plus five maintainer children. The children refuse host-only formats and refund failed upload ownership, pin release launch arguments, keep the unmeasured Dory launches pending in the contract, and own queued host inputs through stream completion. Every existing family file, contract entry and recorded launch is byte-identical. Thank you, Jürgen. On a GB10 with nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4 @ 41f02335, served at 2 loops with CUDA graphs:
- Synthema, Ash's AI Assistant. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Step 3, third kernel PR of the Dory family (#1233). Two activation rows, a block a row, 16-byte loads, fp32 scale vectors formed on the host as torch forms them.
ActRows.gateL2andActRows.swigluL2launch them; no existing launch changes.This PR closes nothing. The issue is #1233.
Stacking. Stacked on #1257 (qk_rows), which is stacked on #1256 (norms, carrying the #1245 contract include fix as its base commit). This branch is one commit on top of #1257.
What the kernels compute
tf_gate_l2_rows:out = bf16(l2(bf16(o * sigmoid(g * sg))))over all heads of a row (dory.py's output gate, thenjustnormovernum_heads * head_dim= 4,096).sg = (s_gate * (sqrt(g0)/std))^2, an fp32 vector.tf_swiglu_l2_rows:out = bf16(l2(bf16(silu(bf16(g * sug)) * bf16(u * suu))))over the MLP width 7,168.su = suv * sqrt(hidden)in fp32.sigmoidandsiluare torch's fp32 forms,1 / (1 + expf(-x))andx / (1 + expf(-x)), with IEEE division.expfoverflow (x < -88.7) gives sigmoid exactly 0 as torch does; the reference models fp32's exp range.Rounding points: the scaled gate, the scaled up, the product, and the gated row before its norm.
Kernel contract
Registered with the kernel contract. Every output is compared with float64's range at its rounding points (expf's ulps, fp32's exp range, the inverse's spread) at Dory's widths 4,096 and 7,168.
expdid not overflow whereexpfdoes; fixed in the reference.tf_gate_l2_rows: 107/109/109/109/109tf_swiglu_l2_rows: 109/118/118/121/121main.Receipt
50b11cf70; NVIDIA GB10 (sm_121), CUDA 13.0 nvcc, Zig 0.17.0.zig build test -Dnvcc=...: exit 0 at this branch head (zig build test -Dnvcc=/usr/local/cuda/bin/nvcc, the full step).main.mainplus the include fix,SUMMARY 280 kernels: 278 pass gates 1-4, 2 fail, 2 known findings; 53 controls: 52 red as planted, 1 missed; 1566 limit problems. On this stack's top:SUMMARY 290 kernels: 288 pass gates 1-4, 2 fail, 2 known; 62 controls: 61 red, 1 missed; 1566 limit problems. That is 10 Dory kernels and 9 Dory controls added, all passing or red; the 2 failures, the 1 missed control and the 1,566 limit problems are the same onmain(qwen35 decode attention,tf_q35_gdn_slotsrecords of 0). Two things onmainkeep the unmodified run from completing:w4a8_tests.zigdoes not compile againstw4a8.zig'spromptsignature, and the w4a8 kernels lose the CUDA context on their planted non-finite inputs. Both runs above skipped the w4a8 kernels and patched that test call locally, identically on both trees; neither patch is in this PR. All Dory launches pass the live limits check.Choices ashhart can change