Repository navigation
lanes: on HIP, Qwen3.6-35B-A3B MTP at four streams falls into a slower, deeper-drafting mode that holds per server process #1695
Description
What happens
With tensorfold-native serve <Qwen3.6-35B-A3B-MLX-4bit-MTP> --parallel 4 on gfx1151, four concurrent 256-token requests at temperature 0 run in one of two modes. The bytes are the same in both. The table below is from the logs of one pair of servers; across all servers, fast requests took 101-120 rounds and slow ones 85-106.
| mode | rounds a request | drafts accepted | decode | aggregate (code) |
|---|---|---|---|---|
| fast | ~115 | 140 of 177-184 | 4.4-4.5 s | 225-239 tok/s |
| slow | 86-106 | 150-169 of 236-262 | 5.0-5.6 s | 179-200 tok/s |
In the slow mode, the lane core drafts deeper, accepts a smaller share, and the rounds cost more than the extra tokens they land. A process usually stays in one mode for all its requests, but some switch between requests.
It happens on main. The runs are tools/bench_concurrent.py --levels 4 --tokens 256 --temperatures 0, 3 or 5 reps a server:
- 11 servers on
main(22f3d28 and 01e25d3, which share the same HIP source): 7 of 51 code reps fell under 215 tok/s, and 1 server's median did. - 12 servers with perf(hip): wide stream rounds reuse cached weights, and a chunk's code pairs are made once #1694: 25 of 56 code reps fell under 215 tok/s, and 6 servers' medians did. That PR makes wide rounds cheaper, which seems to tip the choice more often.
- One stream, which takes the lone-round rule, does not show it.
What I checked
- Disabling the lanes' wall-clock feedback (
h.cfg.timed = falseafterlanes.Config.initinzig/src/native/hip.zig, local experiment only) does not remove the slow mode. Each process then stays in one mode: 1 of 3 servers ran slow throughout (~191 tok/s, 94-101 rounds), and the other 2 ran at ~224 tok/s with 95 rounds every request. - So the choice also depends on the cost tables timed at load, not only on the overhead EMA.
- Turning on
allocate_shared_draftsin HIP's lane facts (.allocate_shared_drafts = drafting, asnative/qwen27_lanes.zigdoes) makes it worse on both sides. Over 3 servers each, 10 of 15 code reps were slow onmainand on perf(hip): wide stream rounds reuse cached weights, and a chunk's code pairs are made once #1694, against 1 of 15 and 6 of 15 without it in the same session.
How the depth is chosen on HIP, as far as I can read it (please correct me):
- Shared rounds go through
Rule.budgetsand thenallocateDrafts(core/lanes/depth.zig,core/lanes/allocate.zig). These maximise expected tokens per ms overshared_costs[rows] + fixedMs(streams) + mtp_step_ms * deepest. shared_costsis timed once at load (hip_serve/costs.zig): 4-row windows at totals 2, 4, 8, 12 and 16, then linear interpolation. The real rounds pad to row buckets 1, 2, 4, 8, 16, 32 and 64.fixedMsis an EMA with weight 0.2, keyed only by stream count (Rule.observeOverhead). There is onelanes.Engineper process, so it is never reset between requests.--mtp-confidence(0.3) is applied in the HIP backend after the core picks a depth. The core does not see the head's probabilities.
My guess, unproven: the overhead term, or the interpolated table, can settle where deeper rounds look cheaper than they are, and then stay there. I don't have the per-round budgets trail from a server, since it is recorded in tests only.
Questions
- Is this a known limit of how HIP prices shared rounds, or worth fixing on the HIP side, for example by timing shared rounds at the served stream count and row buckets?
allocate_shared_draftsas it is doesn't help here. Is there another knob I should try first?
I can attach the server logs, the bench_concurrent JSON and the A/B scripts.
Activity
ashhart commentedon Oct 31, 2026
Thanks @AbrahamCete. The bytes are the same in both modes, so this is a speed problem: 179-200 against
225-239 tok/s.
stream count and the row buckets real rounds pad to looks like the right first step. Your
timed = falserunpoints at the load-time table, not only the overhead EMA.
allocate_shared_draftsoff for HIP, given your numbers.Please attach the server logs, the
bench_concurrentJSON and the A/B scripts. Any fix has to keep tokens equal tomain and show both modes gone over the same number of servers.
AbrahamCete commentedon Oct 31, 2026
Thanks @ashhart. The data is in this gist: https://gist.github.com/AbrahamCete/7edb191a21c00c76abb1dd659fca0d19
It holds the
bench_concurrentJSON and the server log for each of the 32 Qwen3.6-35B-A3B runs behind the numbers here and in #1694, and the five scripts that ran them (local paths replaced by variables). The README lists the binaries, the two local experiments (timed = false, andallocate_shared_draftsin HIP's lane facts) and each server's code median. Token hashes are the same in all 32 runs.I'll start on the fix you describe: timing shared rounds at the served stream count and at the row buckets real rounds pad to. I'll report it against
mainover the same number of servers, with tokens equal.ashhart commentedon Oct 31, 2026
Thanks @AbrahamCete, that's everything we asked for: the JSON, the logs and the scripts for all 32 runs. We'll go through it after 1.2.0, which we're cutting this morning. Tonight's work on how shared rounds price their drafts (a head's step cost was being dropped when drafts are reserved before proposing) may bear on the deeper-drafting mode you see, so we'll check that against your runs first.
AbrahamCete commentedon Oct 31, 2026
/atc claim
AbrahamCete commentedon Oct 31, 2026
/atc presence 1dd3497bef803d2903dc53fa458c1bf5348dc0b136240da710ffaa73290305eb
AbrahamCete commentedon Oct 31, 2026
An update while you cut 1.2.0. Timing HIP's shared rounds at the served stream count and at the row buckets they pad to removes both modes here. Results on
main94b90c9, Qwen3.6-35B-A3B MTP--parallel 4, 6 servers a side, alternated (--levels 4 --reps 5):main:Token hashes are equal on every server. Qwen3.8-27B at 1, 4, 8 and 16 streams is unchanged, and alone and serial are equal.
The cause:
main, most shared totals were priced from single-stream timings.The cost is load time: +1.5 s for the 35B at 4 streams and +20 s for the 27B at 16 streams, which now times the 256-row bucket its widest rounds pad to. I'll rerun this on 1.2.0 with your pricing fix and open the PR then, unless you'd rather see it now.