Repository navigation
glm cuda: load an Apache-2.0 DFlash2 drafter (canada-quant DFlash2-G: 8 full-attention layers, 9 taps)? #1583
Description
Summary
GLM's native DFlash2 drafter pays off on CUDA, but the drafters it loads are incoai/GLM-5.3-Flash-DFlash2 and copies derived from it, CC BY-NC-ND 4.0 (the engine's startup line says so), so they cannot serve commercial work. canada-quant/GLM-5.3-Flash-DFlash2-G is Apache-2.0, trained for the same target, and reports slightly higher acceptance, but it has a different shape and dflash_load.zig refuses it. Would you take a pull request that lets the GLM family load it?
Why it is worth it
Two DGX Spark (GB10, 128 GB), nvidia/GLM-5.3-Flash-FP4-Local @ 76348ddb, main a881b6f, --context 500000 --parallel 4 --kv-cache fp8, TP=2, tools/bench_concurrent.py --levels 1,4 --tokens 256 --reps 3 --temperatures 1.0,0 --alone --serial (0 unequal in both arms), the checkpoint's MTP head against --drafter with incoai's DFlash2, tok/s:
| MTP | DFlash2 | |
|---|---|---|
| code T=0, 1 / 4 streams | 43.4 / 102.3 | 80.3 / 188.7 |
| chat T=0, 1 / 4 streams | 34.2 / 81.9 | 36.0 / 100.1 |
| code T=1, 1 / 4 streams | 46.2 / 63.5 | 50.4 / 67.5 |
| chat T=1, 1 / 4 streams | 36.3 / 54.5 | 38.2 / 64.6 |
At TP=3 with #1576, the drafter also runs: code T=0 56.7 tok/s at one stream and 137.9 at four.
The two drafters
| incoai DFlash2 (loads today) | canada-quant DFlash2-G | |
|---|---|---|
| licence | CC BY-NC-ND 4.0 | Apache-2.0 |
| layers | 5, sliding_attention, window 2,048 |
8, full_attention |
| target taps | 5, 14, 24, 33, 42 | 5, 9, 14, 19, 24, 28, 33, 38, 42 |
| block, conv, selector | 8, kernel 2 group 16, rank 256 top 16 | the same |
| mask token | in the safetensors | id 154856, embedding in mask_embedding.pt |
| publisher's acceptance (K=7, holdout, thinking on, T=1) | 3.602 | 3.676 |
The native constants that refuse it today are dflash.zig's LAYERS = 5 and WIN = 2048, cuda_taps.zig's layers = { 5, 14, 24, 33, 42 }, and dflash_load.zig's config check.
Proposal
- The drafter's layer count and target taps read from its
config.json(bounded at load, refused past the bounds), with the tap capture following them. - Full-attention layers served through the existing ring at a fixed window (2,048, as now). Drafts are verified, so a window only moves acceptance, never output; at about 32 KB a token a stream, a true full context would not fit four 500K windows.
- The mask embedding read from
mask_embedding.ptthrough the allow-listed pickle state-dict reader the engine already has for Drama'scodec.pth, or, if you prefer, refused unless converted to safetensors by a documented step. - Receipts:
bench_concurrent --alone --serial(drafted equals"draft": false, concurrent equals solo), resumed against fresh, speed and acceptance against MTP and incoai's drafter at TP=2 and TP=3, and the host tests.
Questions
- Would you take it?
- Is a fixed window for a full-attention drafter acceptable, or would you rather it be refused until a full ring fits?
.ptthrough the allow-listed reader, or a conversion step?
Activity
gilby commentedon Oct 29, 2026
/atc claim
ashhart commentedon Oct 30, 2026
Thanks @gilby. A drafter that can serve commercial work is worth having, and we'll take it.
config.json, bounded at load, and refused past the bounds.acceptance, never output. Please say so at start (the window used, against the drafter's full attention), and
report acceptance against the publisher's figure.
codec.pth.Refuse anything in the file outside the one tensor it needs.
The receipts you list are the gate: drafted equal to
"draft": false, concurrent equal to solo, resumed equal tofresh, at TP=2 and TP=3. Please also show that incoai's drafter loads and drafts exactly as before.