Repository navigation
qwen3_5 cuda: dense EXL3 checkpoints (Qwen3.8-27B), with DFlash2 drafts - #1465
Conversation
|
RTX 3090 (sm_86) receipt for this branch, commit c127b64. Setup. RTX 3090 24 GB, Linux. Image built from this commit with Load. The server log says Exactness. One greedy chat: 33 prompt tokens, 76 completion tokens. Drafts on and Decode, same request, wall time, so the short prefill is included:
A repeat of that short prompt reported Not run. No context above 16384, no long-prompt prefill, no teacher-forced compare against ExLlamaV3. |
|
Thanks, this is the split we hoped for. It only admits a dense EXL3 checkpoint that main refuses and runs it on the existing I'll repeat your exactness receipts on one of our GB10s next: the dense pack with DFlash2 for drafted == plain, concurrent == solo at levels 1 and 4 with temperatures 0 and 1.0, and resumed == fresh by token hash with prompt tails 16 and 17 rows past a kept state; plus a MoE EXL3 checkpoint and the MLX 27B against main. Two small questions. The resumed == fresh receipt for the 38.7k two-turn chat says both answers are correct, not that they are equal. And "the 8k chat repeat resumed at 12,288" can't be right for an 8,020-token prompt. Could you check those? Not blocking, but worth a follow-up: the licences of the turboderp pack and the z-lab drafter in the recipe, and the recipe's EXL3 section, which still describes the Python engine's span-sliced head. |
…t plans as the whole head does The 384-column tail planned eight K splits where the full head plans one, so its logits and some proposals changed. The view now starts at column 240,128 (exl3.wide_blocks blocks), keeping ashhart#1465's bits. A host test checks that both span views plan as the whole head does.
The EXL3 reader admitted Qwen3.5-MoE checkpoints only. A dense checkpoint's MLP now loads as three Dense projections (gate, up, down) beside the attention, delta and head projections the reader already loads, and the grouped experts' scratch is sized only for MoE. A dense EXL3 checkpoint drafts as an MLX one does: the DFlash2 head already projects through Dense. MoE EXL3 checkpoints still refuse a drafter. No kernel or arithmetic changes.
c127b64 to
b78c7af
Compare
…o vocabulary spans On ashhart#1465 the drafter projects all 248,320 columns of an EXL3 target's head. It now projects only the spans it proposes from, as column views of the stored matrix: exl3.Weight.columns keeps the parent's strides and moves to whole 128-column blocks, with a host test. The x3 arm shares the affine head's packing path. MoE EXL3 targets still refuse a drafter. The family's formats list exl3, and exl3.md and cuda.md no longer say the native engine serves no EXL3 pack.
…t plans as the whole head does The 384-column tail planned eight K splits where the full head plans one, so its logits and some proposals changed. The view now starts at column 240,128 (exl3.wide_blocks blocks), keeping ashhart#1465's bits. A host test checks that both span views plan as the whole head does.
…o vocabulary spans On #1465 the drafter projects all 248,320 columns of an EXL3 target's head. It now projects only the spans it proposes from, as column views of the stored matrix: exl3.Weight.columns keeps the parent's strides and moves to whole 128-column blocks, with a host test. The x3 arm shares the affine head's packing path. MoE EXL3 targets still refuse a drafter. The family's formats list exl3, and exl3.md and cuda.md no longer say the native engine serves no EXL3 pack.
…t plans as the whole head does The 384-column tail planned eight K splits where the full head plans one, so its logits and some proposals changed. The view now starts at column 240,128 (exl3.wide_blocks blocks), keeping #1465's bits. A host test checks that both span views plan as the whole head does.
|
Thank you @BHCC2025. This is in TensorFold 1.2.0, released today: https://github.com/ashhart/TensorFold/releases/tag/v1.2.0 It landed with your commits under your own name, so you're listed as a TensorFold contributor, and it's credited to you in the release notes and the changelog. |
|
Thank you, and thanks for the credit. Following up on your two questions and the follow-ups:
|
What this changes
The CUDA EXL3 reader (#1150) admits Qwen3.5-MoE checkpoints only, and refuses a drafter on any EXL3 checkpoint. This lets it read dense ones too: turboderp's
Qwen3.8-27B-exl3packs, withz-lab/Qwen3.8-27B-DFlash2drafts. It is the dense EXL3 path that #603 asked for and that was left with #832.cuda_geometry.zig: an EXL3 config is admitted dense or MoE (learned neurons are still refused).cuda_exl3_weights.zig: a dense layer's MLP loads as threeDenseprojections (gate_proj,up_proj,down_proj), beside the attention, delta and head projections the reader already loads. The grouped experts' scratch is sized for MoE checkpoints only.cuda_native.zig: a dense EXL3 checkpoint opens DFlash2 or MTP as an MLX one does; MoE EXL3 checkpoints still refuse a drafter. The DFlash2 head already projects throughDensefor EXL3 (cuda_draft_frame.zig).docs/recipes/qwen3.8-27b.md: the native CUDA format list. It said the engine reads no NVFP4 or EXL3 checkpoint, which was out of date for NVFP4 as well.No kernel, launch or rounding changes. Every projection runs on the
cuda.exl3.Densepath that the MoE checkpoints already use.Receipt
Environment.
mainatb30d5b9dplus this commit. One DGX Spark (GB10, sm_121, 128 GB unified memory), Ubuntu 24.04.5, driver 580.178.04, nvcc 13.0.88, Zig 0.17.0,zig build install native -Dsm=121.turboderp/Qwen3.8-27B-exl3@113cf7ab(the4.00bpwbranch:mul1codebook, 6-bit head; both shards' sha256 match the revision's LFS ids).z-lab/Qwen3.8-27B-DFlash2@50307d4c.--context 131072; greedy unless stated.Before.
mainrefuses the checkpoint at load:the native CUDA engine cannot load ... with its own glue kernels (UnsupportedQwenQuantization).Exactness.
tf-qwen35-cuda <model> step|window|treeover 48 ids: window logits equal step logits byte for byte, and so do tree logits.bench_concurrent.py --alone --serial --levels 1,4 --temperatures 0,1.0(drafts on): alone 20/20 equal, serial 20/20 equal, 0 failed."draft": false: an 8,020-token chat prompt, 256 reply tokens, all three runs give the same reply hash (fresh, then the cached repeat, then"draft": false).bench_cache.py --require-reuse --prompt-filedoes not qualify here, because the prompt cache keeps no state for raw token-id completions (kept 0, the same policy for every checkpoint). Its 6 rows still give equal tokens./healthstaysok.Decode (
bench_openai.py --tokens 256 --reps 3 --temperatures 0).fibonacci-raw71.9 tok/s,gpu-chat-no-think45.8.bench_concurrent.pyaggregate at 4 streams: code 137.4 tok/s, chat 101.2 (greedy); 140.9 and 101.4 sampled.Prompt speed.
bench_cache.py --require-cold --prompt-file,--prompt-cache-gib 0, a fresh server per length. The prompts are exact token arrays of this repository'szig/src/**/*.zig(sorted by path, read in order) under the checkpoint's own tokenizer. All four qualified cold:That is below the MLX 4-bit checkpoint's prompt path, because EXL3 prompt rows go through
Densein 128-row blocks. FasterDenseprompt kernels are a separate change, and we have one to propose next.Tests run.
zig build test: 4,585 passed, 25 skipped, 0 failed (Linux x86_64).zig build test-cuda-host: 3,434 passed, 9 skipped, 0 failed.zig fmt --checkis clean on the changed files.python3 tools/zig/lean_check.py: 428 problems on this branch and 428 onmain, the identical list (inherited, for exampletools/zig/unicode_tables.py:1). Zero new.mainand pass here. Onmainthey fail asformat coexistence: EXL3 admits MoE and dense ...(expected the dense config admitted) anda dense EXL3 config (Qwen3.8-27B's layout) is admitted, with no experts. With this change, 0 fail.Not run, and why.
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.Densepath.Thanks to @briepala for the #603 proposal and to @alexderz and @CanadaApollo6 for the RTX 3090 work there. If either of you can run this branch on sm_86, it would close the platform gap above.