Repository navigation
Conversation
This was referenced Nov 6, 2026
Owner
|
Thanks @laudney. The step 1 read matches the plan:
Four small things:
|
Contributor
Author
|
Thanks @ashhart. All four are in
#2384 has the same two commits picked with the same patches, so a rebase onto this PR drops them. |
6 of 8 tasks
Owner
|
Thanks @laudney. Asks 2 to 4 are in: |
Contributor
Author
|
Thanks @ashhart. Here is the sm_121 SASS (nvcc 13.0, the same
The only floating-point reorder is the two independent
|
Clef-Flash ships its Qwen3.5 backbone as the publisher's BF16 weights with no quantization block at all, which the CUDA geometry refused. A dense config with neither an MLX table nor a quantization_config now parses as bf16, its kinds from cuda_modelopt.plain: every used module and the untied head bf16 on the ModelOpt loader's HF names; tied, MoE and pocket configs refuse. The decision-only refusals follow.
Clef-Flash's last shard holds its 0.85 GiB vision tower beside text tensors, which a whole-file count charges to a text-only engine. The CUDA preflight now asks an optional family hook, textWeightBytes, before tensorWeightBytes or the file count; Qwen3.5 answers it for a plain BF16 checkpoint with the stored bytes of its language model and lm_head tensors, and null for every other format.
native/cuda.zig sat at the 600-line lean limit, and the decision-only BF16 entry needs one more line in the capability list. The registry and the families list move whole into cuda_families.zig with their two tests; cuda.zig takes the registry from it and re-exports the list, and drops the family imports only the registry used.
A plain BF16 Qwen3.5 checkpoint is a decision backbone, as on HIP: it serves only with its joint schema head and none of the flags the head is not qualified with, and the head needs the BF16 weights. The CUDA open now checks that before any device work and words each refusal, and refuses a headed checkpoint too until the head's CUDA operators land. capabilities lists BF16 under decisions for qwen3_5.
A Clef readout reads every prompt row after the final norm, while a CUDA prompt chunk runs only its last row past the last layer and then the LM head. Engine.Keep gains rows, a prompt keep that runs every row through the last layer and the final norm into x and stops before the head, without a wait or a recurrent completion mark. tf-qwen35-cuda writes such rows and checks them: on the GB10 the last row equals the .last chunk's, and pieces of 16 to 512 rows give identical bytes.
The CUDA loader stored HF norm weights as bf16(1 + w), which drops w's low bits in every norm; HIP and torch scale by 1 + w in FP32. A plain BF16 checkpoint now keeps HF's raw w and runs tf_q35_norm_shift and tf_q35_head_norm_shift, which add 1 in FP32; other formats keep their kernels and bits. On the GB10 the final rows' mean error against FP32 drops from 2.84e-2 to 2.60e-2 (transformers BF16: 2.49e-2). Both kernels carry contract gates, controls and nvcc 13.0 records.
Gate 1 compares the native final-norm rows of the 40 Clef records with transformers' BF16 and FP32 rows. clef_backbone_rows.py native takes --backend cuda: it runs tf-qwen35-cuda's rows mode per record and saves the rows as the HIP runner's uint16 .npy, so compare is unchanged. On the GB10 every bucket's mean error is within 0.4% of transformers BF16's or nearer FP32, top-1 never differs, and pieces of 512, 1,024 and 4,096 rows give identical bytes.
The first step of ashhart#2324 lets the CUDA Qwen3.5 engine load Clef-Flash's BF16 backbone for decisions only; the Unreleased notes say what it loads, what it refuses before the weights load, and how capabilities lists it.
The CUDA Qwen3.5 family listed bf16 under capabilities' decisions while every BF16 open is still refused with QwenJointHeadNotServed, so a client read a format it cannot use. Drop decision_formats and its family wiring until the joint schema head serves in step 3.
The review asked the line to end with the PR number and the thanks, and to say that a quantized checkpoint shipping the joint head config is now refused as on HIP; the capabilities note leaves with the listing, which moves to the head step.
Owner
|
Thanks @laudney. Main had moved under two lines, so I rebased the branch onto it (
Your commits are unchanged otherwise, and the landing gate's host checks pass on the rebased head. It's in our landing queue now. |
ashhart
force-pushed
the
pr-clef-cuda-1
branch
from
November 7, 2026 01:49
7311c7e to
36618c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
servethe checkpointUnsupportedQwenQuantization(no plain-BF16 path)QwenJointHeadNotServed) until plan step 3joint_head_config.jsonQwenBf16DecisionOnly)--drafter,--slide,--vision,--kv-cache fp8,--prompt-precision int8on it--draftergives "--drafter: a BF16 Qwen3.5 decision backbone drafts nothing; serve without --drafter"joint_head_config.jsonQwenHeadNeedsBf16), as on HIPcapabilities --json"decisions": {}"decisions": {}; the BF16 listing comes with plan step 3, where/v1/systemoneservesWhat this changes
Plan step 1 of #2324: the CUDA Qwen3.5 engine loads the publisher's plain BF16 checkpoint as a decision backbone, as on HIP (#2046), and refuses chat and every flag the joint schema head is not qualified with by a named error before any device work. Its norms scale by 1 + w in FP32 (HF's raw Qwen3.5 norm weights), and a readout piece can keep its final-norm rows with no head (
Keep.rows), which plan step 2 consumes./v1/systemone, step 4 the GB10 gates 1-6).0d7152955. Commitsc35c3f17a..7311c7ef8:c35c3f17aparse a plain BF16 Qwen3.5 config (Geometry.bf16; every used slot and the head in theqmmfBF16 lane; tied or MoE refused)726747a92count its text tensors at preflight (textWeightBytes; the vision tower is not counted)d62a13484refactor: the CUDA family registry moves tonative/cuda_families.zig(native/cuda.zigwas at the line limit)3aca0ab99the named refusals (cuda_decide.zig) anddecision_formats = {"bf16"}(taken out again by0be7071d2)200b2f24eKeep.rows: every row through the last layer and the final norm, no head; shape key bit 3010c254b7dtf_q35_norm_shiftandtf_q35_head_norm_shift(1 + w in FP32) for a BF16 checkpoint; MLX, ModelOpt, compressed-tensors and EXL3 keep the old kernels58d77dc3ctools/clef_backbone_rows.py --backend cudareads the rows throughtf-qwen35-cuda8f60977cfCHANGELOG line citing Clef-Flash on native CUDA: /v1/systemone through the joint schema head (GB10) #23240be7071d2review: nodecision_formatson CUDA until the head serves (plan step 3)7311c7ef8review: the CHANGELOG line names theQwenHeadNeedsBf16refusal and ends with the PR and thanksatc overlap --json(pr-clef-cuda-1, 2026-11-06 16:25 UTC; incomplete:otherPullRequestsLimit,pullFilesLimit): conflicts with cli: tensorfold models --json, the listed checkpoints as one document #2364, server: a refused API key file logs the path and the mode it read #2365, qwen27 metal: a keep that fails part way ends its encoder, so the error returns instead of aborting the process #2376 and docs: an optimization ledger, which speed and memory change applies to which family #2232 inCHANGELOG.mdonly, and with fix(cuda): allocate Qwen3.5 scratch lazily #1879 (lazy Qwen3.5 scratch) in two files:cuda_engine.zig, one line (record = keep.verifies()here,keep != .lastthere), andnative/cuda.zig, where both add a test block at the end. Whichever lands second keeps both sides. Against main5daafaf18the merge is clean exceptCHANGELOG.md, andregistry.zigmerges to 599 lines.core/lanes/change. No Clef code is ported in this step, so THIRD_PARTY_NOTICES is unchanged; the head kernels (step 3) add their files to the "Cloudflare Clef" entry.Checks on this head
7311c7ef8. Engine code is58d77dc3c, which the receipts ran, less the CUDAdecision_formatslisting (0be7071d2), which no open path reads.0d7152955and on this head: 252 and 252, 0 new.zig build test-cudawith the device tests): 565/565 steps, 7339/7400 tests passed, 61 skipped.zig build test test-golden(host, Linux x86_64): first run 7353/7414 passed, 60 skipped, 1 failed intest-cuda-host; the failure did not repeat: the two binaries it reported pass alone with the same seed0xd7652ec0(114/114; 1,110/1,112, 2 skipped), and a second full run exits 0.7311c7ef8:zig build test-cuda-host test-qwen35-modelopt-admission-hostexits 0.zig build test-qwen35-modelopt-admission-host: 117/117, with the Clef-shaped BF16 configs (decision-only, every refusal, tied and MoE refused, slots in the BF16 lane).tf-cuda-test qwen-norm-shift: both shift kernels within one BF16 ulp of FP64 (1 + w) x inv over 5 rows of 4,096 and 30 heads of 256; the unshifted kernels on raw w fail the same gate (controls red); the gate rejects NaN and inf on its own fixture first.tf-qwen35-cuda <clef> rows-lastandrows-cuts: the last row ofKeep.rowsequals the head path's final-norm row; pieces of 16 to 512 give 0 differing rows.zig fmt --check build.zig zigandgit diff --check: pass.pull_requestworkflow while the PR conflicts with main, and the only conflict isCHANGELOG.md(main8bf5dbc35; the mergedregistry.zigis 599 lines).Receipt
Environment: base
0d7152955, head8f60977cf(receipt binaries from58d77dc3c); Zig 0.17.0; CUDA 13.4 for the builds and runs, nvcc 13.0 for the launch records; NVIDIA GB10 (sm_121, 128 GB shared memory, Linux aarch64); checkpoints Cloudflare/clef-flashfb4417d31c1d0f9e7e77f1141b40aaede7491e00and TensorFold/Qwen3.8-27B-MLX-4bit22d8d538154e0e5b6f8dcb1fcc74a60a0798104c.Paths and platforms: the CUDA Qwen3.5 engine. A BF16 checkpoint takes the new parse, preflight count, refusals and shift norms; every served CUDA Qwen3.5 checkpoint (MLX, ModelOpt NVFP4, compressed-tensors, EXL3) passes through the same
open,forwardandfinish, so the 27B rows below cover the existing path. Ran: GB10. Not run: other CUDA GPUs (records only).Rows vs transformers,
tools/clef_backbone_rows.pyover 40 records of the public corpus, pooled mean abs error of the final-norm rows against transformers FP32:The top-1 next token never differs from transformers BF16's. Without the 1 + w fix the 3,000-token probe read 2.84e-2 (BF16 2.49e-2); with it, 2.60e-2. The GDN chain stays in its BF16 lanes; the 4,097-10,240 band is 0.34% further from FP32 than transformers BF16.
Exactness,
tools/bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial, 27B, runs in the order main, this PR, the readouts PR, the readouts PR, this PR, main: every cell's concurrent, alone and serialtoken_sha256agree within each run and across all six runs (code2b28aa2108de…, chata06fab348fbe…).Main against this branch (27B,
--prompt-cache-gib 0 --context 70000, one server per run):bench_openaidecode median, fibonacci-raw / gpu-chat-no-thinkPrompt speed, cold, best of 3 (one request, fresh KV):
tools/bench_cache.py --require-cold, 32k prompt, first requestttft_s: main 26.33 s and 26.30 s, this PR 26.31 s and 26.32 s.Licence of new weights: n/a (no weights shipped; the receipts read Cloudflare/clef-flash under its own licence).
Kernel contract
Kernels registered:
tf_q35_norm_shiftandtf_q35_head_norm_shift(contract_q35_norm_shift.zig: the residual-add norm at 4,096 and the q/k head norm at 256, with and without residual), besidetf_q35_normandtf_q35_head_norm, which they do not replace (every non-BF16 checkpoint keeps those).Planted controls: each shift kernel's runner plants the unshifted weight; both red.
SUMMARY 592 kernels: 592 pass gates 1-4, 0 fail, 8 known findings; 373 controls: 373 red as planted, 0 missed; 689 limit problems; gates 3261206 ms, limits 962691 ms, exit 1. The 689 limit problems are the CUDA 13.4 build checked against records from nvcc 13.0; the records check is the 13.0 one below.Records from nvcc 13.0 (
zig build fatbins kernel-contract-fatbins -Dnvcc=…/cuda-13.0/bin/nvcc -Dsm=80,86,89,120,121, thentf-cuda-test kernel-contract-limits zig-out/fatbin):tf_q35_norm_shift54/55/55/48/48 registers, 16,416 static shared bytes, bound 256;tf_q35_head_norm_shift24/24/24/22/22 registers, 16 bytes, bound 64; every new record current.FAIL kernel-contract-limits: 226 problemson this head and 226 on main: all inherited.Chips run: GB10 (sm_121) for gates 1-4 and the controls. sm_80, sm_86, sm_89 and sm_120: limits only.
The two shared kernels,
tf_q35_normandtf_q35_head_norm, now wrap the templates withSHIFT = false. Main0d7152955against this PR's kernel source (58d77dc3c, unchanged since), both from one nvcc 13.0 build (zig build fatbins -Dnvcc=…/cuda-13.0/bin/nvcc -Dsm=80,86,89,120,121;qwen35.fatbinis the only fatbin with these kernels). SASS hash: sha256 (first 16) of thecuobjdump -sass -arch sm_XX -fun <kernel>instruction lines. Registers and static shared fromcuobjdump -res-usage:tf_q35_norm48540065e5bc0ce1tf_q35_norm6e1f640ee37e6e27tf_q35_norm834eb1a3c5c15bdatf_q35_norm077f28021fc3f4efd80d6041e8161223tf_q35_head_norm86ed40157f9efdbetf_q35_head_norm1cd8821dc3a5b290tf_q35_head_norm361c32b3f79d0e23tf_q35_head_norm525e1cb20d4359d46189ad939c62b68atf_q35_norm's record (54/55/55/48/48, 16,416) holds on both;tf_q35_head_normhas no launch record on main. At sm_120/121 the SASS differs in register names, integer address arithmetic (IADD3forLOP3on the same offsets) and, intf_q35_head_norm, the order of its instructions; the floating-point instructions are the same in kind and count, and intf_q35_normthe floating-point and memory sequence is the same. To show the bytes, a driver-API harness loads bothqwen35.fatbinfiles on the GB10 and runs each kernel on the same random BF16 inputs:tf_q35_normat D 1,024 to 8,192 (both the 8-wide and the scalar paths) with and without the residual, the group sums and the row count, at two input scales;tf_q35_head_normat 4, 16 and 32 heads, packed and strided, with and without the row count.PASS 152 of 152 cases byte-identical (74103680 output bytes compared); its control (the second fatbin run witheps2e-6) givesFAIL 76 of 152. sm_120's SASS hash equals sm_121's on both sides, so the GB10 run covers it.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.--drafter, no slower than without it, greedy and sampled: 1 and 4 streams measured (8 admitted, not run); no drafter run, because the 27B path takes no new branch with one