[gilby]

Clef-Flash as a runtime adapter on Qwen/Qwen3.5-9B, like the 27B Clef of #1364: one server answers chat with the base weights and POST /v1/systemone with Cloudflare's Clef-Flash applied to decision pieces only. Stacked on #1364 (runtime adapter), which is stacked on #1361 (Clef endpoint and head, issue #1360); the only new commit here is the last one.

Change

The qwen3_5 CUDA family reads GDN norm weights as bf16, and stock Qwen/Qwen3.5-9B stores its 24 linear_attn.norm.weight tensors in f32, so the stock checkpoint is refused (… f32 { 128 }, expected bf16 { 128 }). This PR keeps that refusal and adds an explicit conversion step instead of rounding at load:

python3 tools/qwen35_bf16_norms.py Qwen3.5-9B/ Qwen3.5-9B-bf16norms/

• Each f32 *.linear_attn.norm.weight is rounded to bf16 (nearest, ties to even, as torch's cast; bit-identical to torch on 200,006 values); every other tensor is unchanged (A_log stays f32, which the family reads as f32). Cloudflare's merged release stores these norms rounded the same way.
• Shards without such a norm are hard-linked (copied where links fail); the index's total size is rewritten. Standard library only; it streams each shard (54 MB RSS, 4.6 s on the 9B). A checkpoint with no f32 norms is refused before anything is written.
• tools/zig/qwen35_bf16_norms_test.py (ties, NaN, only the norms change, other files byte-identical, the refusal) runs in no-gpu.yml.

Nothing else changes: #1364's adapter code already walks every layer and projection by name and checks each factor against the model's widths, so the 9B's dense FFN, GDN and attention projections work as they are.

Adapter weights

Build the adapter directory with #1366's tools/clef_adapter.py --merged <Cloudflare/clef-flash> --base <Qwen/Qwen3.5-9B> --out <dir>: 248 projections differ (every linear projection of all 32 layers, rank 256 or the full rank 32 for the GDN in_proj_a / in_proj_b); the other 48 differing tensors are the f32 A_log / norm weights, equal after rounding to bf16. The residual of each factorisation is at most 1.02× the bf16 rounding floor of the merge.

Receipts (one DGX Spark, GB10 128 GB, CUDA 13.0, -Dsm=121; Cloudflare's 60 records / 557 questions; head 7f1b31c2d)

• Extraction fidelity, Transformers bf16 on an Apple GPU: stock base + adapter (forward hooks) vs the merged release: mean KL 4.6e-5, top choice 99.64% (100% up to 4K tokens), the bf16 noise floor (the merged release on native TensorFold scores 5.7e-5 / 99.64% against the same reference, clef: Cloudflare Clef decisions on /v1/systemone through its joint schema head (--clef DIR) #1361).
• Native, the tool's copy of Qwen/Qwen3.5-9B + --clef <adapter dir>, against Cloudflare's own bf16 output: mean KL 5.20e-5, top choice 99.64% (≤1K 100%, 1–4K 100%, 4–10K 99.6%, >10K 99.4%). Gate (mean KL ≤ 0.05, top choice ≥ 95%): pass.
• Chat replies are bit-identical before and after decision passes ran the adapter; four concurrent decisions equal decisions alone (60/60 records); bench_concurrent --tokens 128 --reps 3 --temperatures 0 --levels 1,4 --alone --serial: 0 unequal, 0 failed.
• Chat with and without --clef, same binary, same session (two rounds each): bench_concurrent --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial gives the same tok/s (code 14.9 / 56.2, chat 14.3 / 54.0 at 1 / 4 streams) and the same token shas; the adapter costs 2.1 GiB at startup (model line 17.61 → 19.75 GiB, the decision cache and the packed adapter).

Decision latency (60 records one at a time, --parallel 4 --context 32768)

#1364 now runs the adapter's two products on the base's bf16 lane GEMM (tensor cores) instead of DenseRows. On this base, where all 32 layers are adapted, that took decisions from about 4× the merged release served on its own to about 1.2×:

tokens	runtime adapter, DenseRows p50 / p95 (s)	runtime adapter, lane GEMM p50 / p95 (s)	merged release alone p50 / p95 (s)
≤1K (22)	1.13 / 2.23	0.40 / 0.70	0.35 / 0.60
1–4K (15)	5.03 / 9.49	1.46 / 2.76	1.22 / 2.32
4–10K (15)	18.80 / 23.99	5.18 / 7.09	4.29 / 5.92
>10K (8)	36.74 / 38.82	10.65 / 11.54	8.80 / 9.67

The rest is the adapter's own work (rank-256 products on every projection, about 7% of the base's multiply-adds, plus a bf16 add per projection).

Checks

• python3 -m unittest discover -s tools/zig -p qwen35_bf16_norms_test.py: 3/3.
• lean_check.py adds no problems; the sm_121 CUDA build succeeds.

[ashhart]

Thanks @gilby. We can't take this as written: bf16vec rounds the stock checkpoint's f32 GDN norm weights to bf16 at load, for every request including chat, and on by default. That's a lossy load, which the engine doesn't do even opt-in. Two ways forward: read the f32 weights through a kernel variant (registered with the CUDA kernel contract, with records), or keep refusing these checkpoints and document an explicit conversion step. The rest of the stack follows #1361 and #1364's changes.

[gilby]

Yup. I didn't like it either, and am having my agent do it better. ;)

We had an awesome setup with Qwen3.8-27B and Clef on the old python branch, where it just loads the weights once, and wanted to bring this to the Zig branch along with the faster 9B one. We'll get'r done

[gilby]

Thanks, agreed: a lossy cast at load is the wrong default. I took the second route. The branch no longer touches the loader (d4559f94d is gone), so stock f32 GDN norms are still refused:

error: model.language_model.layers.0.linear_attn.norm.weight: f32 { 128 }, expected bf16 { 128 }

The conversion is an explicit step instead: tools/qwen35_bf16_norms.py SRC OUT (6ab5c0fd7, stacked on #1364's new head).

• It writes a copy in which each f32 *.linear_attn.norm.weight is rounded to bf16 (nearest, ties to even, the same as torch's cast; checked bit-for-bit against torch on 200,006 values). Every other tensor is unchanged; A_log stays f32.
• Shards without such a norm are hard-linked, and the index's total size is rewritten.
• It uses the standard library only and streams each shard (54 MB RSS on the 9B, 4.6 s).
• It refuses a checkpoint that has no f32 norms, before creating anything.
• Its unittest (tools/zig/qwen35_bf16_norms_test.py: ties, NaN, only the norms change, other files byte-identical, the refusal) runs in no-gpu.yml.

Gate on this head (one DGX Spark, GB10, CUDA 13.0, -Dsm=121): Qwen/Qwen3.5-9B converted with the tool, plus the Clef-Flash adapter, against Cloudflare's 60 records / 557 questions.

• Mean KL 6.92e-5 and top choice 99.46%. Gate passes.
• Decisions are bit-identical (60/60 records) to the earlier round-at-load build and to an earlier converted copy.
• 4 concurrent decisions equal decisions one at a time: 60/60.
• A greedy chat reply is bit-identical before and after the decisions.
• bench_concurrent --tokens 128 --reps 3 --temperatures 0 --levels 1,4 --alone --serial: 0 unequal, 0 failed.

The title and body are updated to match.

[ashhart]

Review against main e9b69ed (shared notes on #1361). bf16vec now accepts F32 GDN norm weights and rounds them to BF16 at ordinary load, with or without --clef (qwen3_5/cuda_weights.zig:146, :245, :385): F32 1.00390625 becomes 1.0. That changes stock Qwen3.5-9B's values, which we can't do even when the result looks close. Two exact options: keep the F32 norm weights and add a matching kernel path for them, or show the stock release itself multiplies these norms in BF16 (its own operation-by-operation casts). Until then the engine should keep refusing F32 norms by name. The adapter part inherits #1364's exactness blocker.

[gilby]

Closing along with #1364: this builds on the runtime adapter, which can't match the merged release bit for bit. Clef-Flash upstream stays the merged-release endpoint of #1361.