Repository navigation
Conversation
Clef-Flash ships its Qwen3.5 backbone as the publisher's BF16 weights with no quantization block at all, which the CUDA geometry refused. A dense config with neither an MLX table nor a quantization_config now parses as bf16, its kinds from cuda_modelopt.plain: every used module and the untied head bf16 on the ModelOpt loader's HF names; tied, MoE and pocket configs refuse. The decision-only refusals follow.
Clef-Flash's last shard holds its 0.85 GiB vision tower beside text tensors, which a whole-file count charges to a text-only engine. The CUDA preflight now asks an optional family hook, textWeightBytes, before tensorWeightBytes or the file count; Qwen3.5 answers it for a plain BF16 checkpoint with the stored bytes of its language model and lm_head tensors, and null for every other format.
native/cuda.zig sat at the 600-line lean limit, and the decision-only BF16 entry needs one more line in the capability list. The registry and the families list move whole into cuda_families.zig with their two tests; cuda.zig takes the registry from it and re-exports the list, and drops the family imports only the registry used.
A plain BF16 Qwen3.5 checkpoint is a decision backbone, as on HIP: it serves only with its joint schema head and none of the flags the head is not qualified with, and the head needs the BF16 weights. The CUDA open now checks that before any device work and words each refusal, and refuses a headed checkpoint too until the head's CUDA operators land. capabilities lists BF16 under decisions for qwen3_5.
A Clef readout reads every prompt row after the final norm, while a CUDA prompt chunk runs only its last row past the last layer and then the LM head. Engine.Keep gains rows, a prompt keep that runs every row through the last layer and the final norm into x and stops before the head, without a wait or a recurrent completion mark. tf-qwen35-cuda writes such rows and checks them: on the GB10 the last row equals the .last chunk's, and pieces of 16 to 512 rows give identical bytes.
The CUDA loader stored HF norm weights as bf16(1 + w), which drops w's low bits in every norm; HIP and torch scale by 1 + w in FP32. A plain BF16 checkpoint now keeps HF's raw w and runs tf_q35_norm_shift and tf_q35_head_norm_shift, which add 1 in FP32; other formats keep their kernels and bits. On the GB10 the final rows' mean error against FP32 drops from 2.84e-2 to 2.60e-2 (transformers BF16: 2.49e-2). Both kernels carry contract gates, controls and nvcc 13.0 records.
Gate 1 compares the native final-norm rows of the 40 Clef records with transformers' BF16 and FP32 rows. clef_backbone_rows.py native takes --backend cuda: it runs tf-qwen35-cuda's rows mode per record and saves the rows as the HIP runner's uint16 .npy, so compare is unchanged. On the GB10 every bucket's mean error is within 0.4% of transformers BF16's or nearer FP32, top-1 never differs, and pieces of 512, 1,024 and 4,096 rows give identical bytes.
The first step of ashhart#2324 lets the CUDA Qwen3.5 engine load Clef-Flash's BF16 backbone for decisions only; the Unreleased notes say what it loads, what it refuses before the weights load, and how capabilities lists it.
A Clef request reads every prompt row through its joint schema head, so the CUDA Qwen3.5 lanes now take readout streams: each prefill step runs one piece from row 0 (no prefix store or kept states) as a rows forward and hands its final-norm rows to the attached readout, release closes the state after the stream drains, and readout_finish reaches the readout through the CUDA host. A piece is the lane host's cap or 1,024 rows, as on HIP. tf-qwen35-cuda readout checks the rows on the GB10: four together, a cancel and a failed consume match one alone byte for byte.
The second step of ashhart#2324 gives the CUDA Qwen3.5 lanes readout streams; the Unreleased notes say how a readout prompt is filled and how large a piece is.
Owner
|
Thanks @laudney. Step 2 reads clean:
One small thing: please end the CHANGELOG line "(#2384, part of #2324). Thanks to @laudney." This lands after #2383. |
The CUDA Qwen3.5 family listed bf16 under capabilities' decisions while every BF16 open is still refused with QwenJointHeadNotServed, so a client read a format it cannot use. Drop decision_formats and its family wiring until the joint schema head serves in step 3.
The review asked the line to end with the PR number and the thanks, and to say that a quantized checkpoint shipping the joint head config is now refused as on HIP; the capabilities note leaves with the listing, which moves to the head step.
The review asked the line to end with the PR number, the issue it is part of and the thanks, as other lines under Unreleased do; the text of the change itself stays the same.
Contributor
Author
7 of 8 tasks
6 of 8 tasks
Owner
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stream.readout)prefill_step; each piece's final-norm rows go to the attached readout on the engine streammax_rowswhen above 0, else 1,024 (HIP's piece); no new settingreadout_finishnative/cuda_bound.zigforwards itconsumeWhat this changes
Plan step 2 of #2324: the CUDA Qwen3.5 lane backend takes readout streams, as HIP does since #2047. A readout prompt fills from row 0 (no prefix store, kept states or reuse marks) through
prefill_steppieces inforward(.rows)(from #2383), and the readout consumes each piece's rows;readout_finishreturns its logits. No readout is attached in production until the joint schema head lands (plan step 3), so every served path keeps its old vtable./v1/systemone, step 4 the GB10 gates 1-6).pr-clef-cuda-1, head7311c7ef8). This PR addsacf6df734(readouts),8b5f05def(CHANGELOG line citing Clef-Flash on native CUDA: /v1/systemone through the joint schema head (GB10) #2324) andc993c194e(review: the line ends with the PR and thanks).8baa42e60ande70f8e295are qwen3_5 cuda: a plain BF16 checkpoint loads as a decision backbone, chat refused by name (#2324 step 1) #2383's two review commits, picked here with the same patches so the stack has no force-push; a rebase onto qwen3_5 cuda: a plain BF16 checkpoint loads as a decision backbone, chat refused by name (#2324 step 1) #2383 drops them.atc overlap --json(pr-clef-cuda-2, 2026-11-06 16:25 UTC; incomplete:otherPullRequestsLimit,pullFilesLimit): conflicts with cli: tensorfold models --json, the listed checkpoints as one document #2364, server: a refused API key file logs the path and the mode it read #2365, qwen27 metal: a keep that fails part way ends its encoder, so the error returns instead of aborting the process #2376 and docs: an optimization ledger, which speed and memory change applies to which family #2232 inCHANGELOG.mdonly, and with fix(cuda): allocate Qwen3.5 scratch lazily #1879 (lazy Qwen3.5 scratch) in two files:cuda_engine.zig, one line (record = keep.verifies()here,keep != .lastthere), andnative/cuda.zig, where both add a test block at the end. Whichever lands second keeps both sides. Against main5daafaf18the merge is clean exceptCHANGELOG.md, andregistry.zigmerges to 599 lines.core/lanes/change: the readout code is in the CUDA Qwen3.5 lane backend (cuda_readout.zig,cuda_native.zig), plus thereadout_finishforward in the CUDA host wrapper (native/cuda_bound.zig) and one build import for the runner. No Clef code is ported; THIRD_PARTY_NOTICES is unchanged.Checks on this head
c993c194e.0d7152955and on this head: 252 and 252, 0 new.tf-qwen35-cuda <clef-flash> readout <3,000 ids> 4on the GB10:PASS 1 vs 4 together with a text prompt between their pieces, after a cancel and a failed consume: 3000 rows in 3 pieces of 1024, byte-identical (wyhash a526ba2c6c9099b9); each state closed once, exit 0. It fails if the planted consume failure does not fire, if the text stream did not run between pieces, or if a state closes more or less than once.zig build test-cuda-host: 5081/5106 passed, 24 skipped, 1 failed:cuda_disk_test"kept states written to disk read back byte-equal after a restart" (expected 2, found 0), a file this PR does not touch; the same binary and seed pass 5 of 5 runs alone (includes thecuda_boundtest thatreadout_finishis forwarded).zig fmt --check build.zig zigandgit diff --check: pass.pull_requestworkflow while the PR conflicts with main, and the only conflict isCHANGELOG.md(main8bf5dbc35; the mergedregistry.zigis 599 lines).Receipt
Environment: base
0d7152955, headc993c194e(the readout runner ran this code; the 27B rows ran the build noted under the table); Zig 0.17.0; CUDA 13.4; NVIDIA GB10 (sm_121, 128 GB shared memory, Linux aarch64); checkpoints Cloudflare/clef-flashfb4417d31c1d0f9e7e77f1141b40aaede7491e00and TensorFold/Qwen3.8-27B-MLX-4bit22d8d538154e0e5b6f8dcb1fcc74a60a0798104c.Paths and platforms: readout streams only take the new path, and only when a readout is attached; text streams keep
prefill.run,prefill_stepand the text vtable. Ran: GB10.Exactness,
tools/bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial, 27B, runs in the order main, qwen3_5 cuda: a plain BF16 checkpoint loads as a decision backbone, chat refused by name (#2324 step 1) #2383, this PR, this PR, qwen3_5 cuda: a plain BF16 checkpoint loads as a decision backbone, chat refused by name (#2324 step 1) #2383, main: every cell's concurrent, alone and serialtoken_sha256agree within each run and across all six runs (code2b28aa2108de…, chata06fab348fbe…).Main against this branch (27B,
--prompt-cache-gib 0 --context 70000):bench_openaidecode median, fibonacci-raw / gpu-chat-no-thinkbench_cache --require-cold32kttft_sThe 27B runs used a build of this PR before one change: it still read an unset
TF_QWEN_READOUT_ROWSonce at open, a setting this head removes with no other change to the text path.Licence of new weights: n/a.
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.--drafter, no slower than without it, greedy and sampled: 1 and 4 streams measured (8 admitted, not run); no drafter run, because no text path changes