[diwakergupta]

What this changes

The dense 27B (hidden 5120, 64 layers, no experts) already copies from a stream's root. Beside DFlash2, a copy now also needs lanes.windows.long_match (20) matching tokens instead of 5, so short suffix matches don't preempt DFlash2's drafts. Without a draft model the 27B keeps main's 5, because there copies are its only proposals. This is part (d) of the #1407 split, and it is independent of the others (based on main).

• Lane core. lanes.drafter.Facts gains copy_match (0: the target's). When Drafted wraps a backend, it replaces the target's copy_match with that value. No other drafter sets it.
• qwen3_5. DFlash2's facts (cuda_draft.zig) give copy_match = long_match on the dense 27B, through copyMatch(g), and 0 otherwise. modelFacts (cuda_native.zig) keeps only root_copies. cuda_geometry.dense27 names the shape.
• This applies to the 27B in every format, the MLX checkpoints included.

Receipt

• Environment. RTX 3090 Ti (24 GB, sm_86). TensorFold/Qwen3.8-27B-MLX-4bit rev 22d8d538, bf16 KV cache, --context 32768, thinking on (the template default). With DFlash2, z-lab/Qwen3.8-27B-DFlash2 rev 50307d4c at 4 bits. Both binaries came from zig build native -Dsm=86, one from main (0d51120a) and one from this branch. Each ran twice, alternating main and this branch, on a restarted server for each drafter mode.

• Exactness. tools/bench_concurrent.py --levels 1 --tokens 384 --reps 3 --alone --serial, with the default code and chat cells and --prompts with two long prompts (5,080 and 2,278 tokens):

  • Every alone and serial ("draft": false) check was equal, in both modes on both binaries.
  • token_sha256 of this branch equals main's in 144/144 replies with DFlash2 and 144/144 without.

• Decode speed, per-stream tok/s, runs 1 / 2 (each a median of 3):

  Cell	no drafter: main	no drafter: this PR	DFlash2: main	DFlash2: this PR	DFlash2 change
  code, sampled	47.4 / 47.3	47.3 / 47.3	250.7 / 251.2	254.1 / 254.7	+1.4%
  chat, sampled	48.1 / 48.0	48.0 / 48.0	134.1 / 134.2	134.1 / 134.3	+0.0%
  code, greedy	48.5 / 48.5	48.6 / 48.6	227.7 / 221.2	230.6 / 234.1	+3.5%
  chat, greedy	48.1 / 48.1	48.1 / 48.2	144.2 / 144.2	153.4 / 153.3	+6.3%
  long prompts, sampled	43.0 / 42.9	43.0 / 42.9	121.6 / 122.8	137.9 / 137.9	+12.8%
  long prompts, greedy	44.1 / 44.1	44.1 / 44.1	113.6 / 112.5	133.4 / 134.6	+18.5%

  • No drafter: within ±0.2% of main in every cell, with the same tokens per round, because the copy match is main's 5.
  • These prompts copy little without a drafter (1.00–1.13 tokens a round), so this table doesn't reproduce the GB10's 2k–64k cells. It shows only that the plain path is now main's.

• Prompt speed: n/a. The copy policy does not reach prompt processing.

• Tests. The full checklist suite, plus test-qwen35-copies-host and test-shared-lane-regressions-host: exit 0, 475/475 build steps succeeded on main at 0d51120a. lean_new.py against main: 0 new problems. New and changed tests:

  • Suffix copies: the dense 27B copies from the root after 20 matching tokens beside DFlash2, after 5 alone (qwen35_copies_host.zig). It checks enter_match 5 with no drafter and copyMatch 20 for the 27B and 0 for a MoE shape. On the first revision it fails, since the target's own match was 20.
  • the depth rule reads the drafter's facts, not the target's (drafted_test.zig) now also checks that Facts.copy_match 0 keeps the target's match and 20 replaces it.

• Not run. GB10. Metal: the policy is CUDA-only.

Checklist

• Every token still goes through the lane rounds.
• Drafted output equals "draft": false. Concurrent and resumed paths are unchanged: no prompt or arithmetic change.
• No precision traded for speed.
• No GPU kernel changed.
• Prompt processing unchanged.
• Platforms: CUDA sm_86 only.
• New tests fail before the change, pass after it.
• Comments and docstrings are one line.
• No personal data, local paths or AI attribution lines.

[ashhart]

Thanks @diwakergupta. Moving the policy into modelFacts with a host test is a nice touch, and since a copy only changes which rows are proposed, the tokens can't change. Merged onto current main it's clean, and fmt, lean, test-qwen35-copies-host, test-shared-lane-regressions-host, test-cuda-host and the CUDA cross-compile all pass.

One thing for the A/B. copy_match becomes the lane core's enter_match (zig/src/core/lanes/config.zig:134), which gates every copy. That includes the context copies the 27B makes when no drafter is attached, and in that case copies are its only proposals. Your receipts are all with DFlash2, so we'll add the same cells without a drafter. If those are slower, the 20-token rule should apply only when DFlash2 is attached.

On a GB10 we'll alternate the default MLX 27B against main: bench_concurrent at levels 1 and 4, code and chat, greedy and sampled, with DFlash2 and with no drafter, plus your two long prompts. Each cell checks token hashes against main and against "draft": false.

[ashhart]

Thanks @diwakergupta. Here are the GB10 results on the default MLX 27B (Qwen3.8-27B-MLX-4bit), main against this PR, alternating main, PR, PR, main:

With DFlash2 loaded: the tokens match main in 20 of 20 prompt rows. Decode after a 2k, 16k and 64k prompt is up 3.0%, 2.5% and 21.5%, and the stream sweep is up 1.4% at one stream and 2.4% at four. Prefill is within 0.6%.

Without a drafter (the default): the tokens again match in 20 of 20 rows, but decode after a prompt drops wherever short copies used to hit. For the four 2k prompts, main decodes at 23.4, 14.2, 14.7 and 23.5 tok/s, and this PR at 13.8 on all four. The median decode after a prompt is down 27.6% at 2k, 29.1% at 16k, 5.7% at 32k and 31.3% at 64k, and unchanged at 8k. The sweep is down 1.8% at one stream and 0.7% at four, and prefill is unchanged.

With no draft model loaded, short copies are the 27B's only proposals, and a 20-token match leaves out most of them. Could the 20-token match apply only when a draft model is loaded, and stay at 5 without one? Then both cases should improve or hold, and we'll rerun both cells on a GB10. This won't make 1.1.8, which ships today, but it can go into the next release.

[diwakergupta]

Thanks @ashhart, that's a clear result. Pushed f8bd7c23 on current main. The 20-token match now applies only when DFlash2 is attached:

• lanes.drafter.Facts gains copy_match (0: the target's), and Drafted.facts uses it in place of the target's copy_match.
• DFlash2's facts give long_match on the dense 27B (cuda_draft.copyMatch). modelFacts keeps only root_copies, so with no drafter the 27B stays at main's 5.
• Tests:
  • The copies host test now checks enter_match 5 alone and copyMatch 20 for the 27B and 0 for a MoE shape.
  • drafted_test's facts test checks that 0 keeps the target's match and 20 replaces it.

On the 3090 Ti (MLX 27B, main 0d51120a against this branch, alternating twice):

• No drafter: 144/144 hashes equal main's, and every cell is within ±0.2% with the same tokens per round. These prompts copy little without a drafter, though, so your GB10 cells are the real check there.
• With DFlash2: 144/144 hashes equal, drafted == plain in every cell, and the same gains as before: +0% to +18.5%, with the long prompts +12.8% sampled and +18.5% greedy.

The full suite: 475/475 steps, and lean shows no new findings. Details are in the description.

[ashhart]

Thanks @diwakergupta, that's the change we asked for. With copy_match in the drafter's facts, the 27B keeps main's 5 with no drafter and takes 20 only beside DFlash2. The two new host tests pin both cases. We've queued the same two GB10 cells on f8bd7c23, plain and with DFlash2, alternating main, PR, PR, main, and we'll post both here.

[diwakergupta]

Hi @ashhart, checking in on the two GB10 cells you queued on f8bd7c23. Any results yet? Also, CI hasn't run on that push: the workflow runs are at action_required and need a maintainer's approval.

[ashhart]

Thank you @diwakergupta. This is in TensorFold 1.2.0, released today: https://github.com/ashhart/TensorFold/releases/tag/v1.2.0

It landed with your commits under your own name, so you're listed as a TensorFold contributor, and it's credited to you in the release notes and the changelog.