[AbrahamCete]

What this changes

Fixes #1695, and is the pricing fix #1694 waits on. On HIP, Qwen3.6-35B-A3B MTP at four streams fell into a slower, deeper-drafting mode that held per server process. The lane core prices shared rounds from the cost table HIP times at load, and that table did not match the rounds HIP runs:

• Shared totals were timed only at 2, 4, 8, 12 and 16 rows, as 4-row windows on at most 8 streams. Every other total up to 16 took a single-stream window's cost.
• Past 16 rows, the lane core scaled the 16-row cost linearly. A real round pads to the next row bucket (round_shape.zig), so a 65-row round ran the 128-row bucket but was priced at about a third of that.

The changes, one commit each:

• fix(qwen35/hip): time shared rounds at the served streams and their row buckets
  • hip_serve/costs.zig gets buckets(). It times one forward for each row bucket up to batch_rows, at the served stream count, with the bucket's rows split evenly over the streams.
  • Every total in a bucket is priced at that forward's cost: a step, not an interpolation. When the table fills, its last entry is the bucket's last total, so the lane core keeps the step.
  • hip_costs.zig sizes its rig for the served streams; the old cap was 8. hip_lanes.zig passes the window width.
  • core/lanes is unchanged, and allocate_shared_drafts stays off for HIP, as asked in lanes: on HIP, Qwen3.6-35B-A3B MTP at four streams falls into a slower, deeper-drafting mode that holds per server process #1695.
• fix(qwen35/hip): time forwards of eight rows or more once
  • Timing every bucket at the served streams added about 20 s to a Qwen3.8-27B load at --parallel 16.
  • From 8 rows on, one pass reads within about 2% of the five-pass mean on both checkpoints, and the first pass (the graph capture) costs the same as the rest. So forwards of 8 rows or more now run once warm and once timed; shorter ones keep 3 and 5.
  • That also covers the lone-window timings main already ran, so both checkpoints now load faster than on main.
• fix(qwen35/hip): time each shared bucket at its own rows up to batch_rows (from review)
  • buckets() capped the timed rows at streams × width. With batch_rows past that cap, the table stopped there and the lane core priced larger totals linearly.
  • Each bucket is now timed at its own rows up to batch_rows, split evenly over min(streams, bucket) windows, so the 128-row bucket at 4 streams runs 4 × 32. width only fed the cap, so it left buckets(), forwards() and measure(), and hip_lanes.zig is back to main.
  • The server sets batch_rows = streams * Hip.max_window (engine/native.zig:210), so no served table changes.

The cost table only steers draft depth, so tokens are unchanged.

Receipt

• Environment: Ryzen AI Max+ 395 (gfx1151), Ubuntu 26.04, Zig 0.17.0, hipcc with HIP 7.1.52801 and Ubuntu clang 21.1.8; checkpoints Qwen3.6-35B-A3B-MLX-4bit-MTP and Qwen3.8-27B-MLX-4bit.
• Base: the branch is three commits on main ba7b2c4. The full A/B below ran on 372af1e, with the first two commits.
  • On ba7b2c4, with all three commits, one server per arm, alternated twice (35B --parallel 4, --levels 4 --tokens 256 --reps 1 --temperatures 0): code 216.5 and 231.3 on main, 237.7 and 237.8 with this PR; chat 193.9 and 193.5 on main, 192.6 and 191.4 with this PR; code 9f2923ac14d0 and chat a8c28b8297b4 on every server; load 13.4 and 13.2 s on main, 10.4 and 10.4 s with this PR. 27B --parallel 16 load: 35.5 and 35.6 s on main, 29.3 and 29.4 s with this PR.
  • On 77179d2, a smoke of 3 servers a side (35B --parallel 4): code 195.8, 213.2 and 230.8 on main, 230.8, 232.0 and 232.9 with this PR; the same token hashes; load 13.8 to 14.0 s against 11.1 to 11.3 s. hip-host-test, test-qwen35-hip-host and check-native-hip-host exit 0.
• The served A/B. Each run is one fresh server, --context 4096 --prompt-cache-gib 0, with the arms alternated server by server.

  • Qwen3.6-35B-A3B MTP, --parallel 4, tools/bench_concurrent.py --levels 4 --tokens 256 --reps 5 --temperatures 0, 6 servers an arm. Code aggregate tok/s, one value a server:

    arm	1	2	3	4	5	6	mean
    main	231.3	205.6	228.3	215.1	227.1	226.9	222.4
    this PR	230.5	232.9	231.7	233.6	231.2	232.0	232.0 (+4.3%)
    this PR with perf(hip): wide stream rounds reuse cached weights, and a chunk's code pairs are made once #1694	242.2	240.2	240.2	241.5	241.5	241.9	241.2 (+8.5%)

    • Chat means: 191.5 on main (182.5 to 194.3), 193.1 with this PR (191.8 to 193.9), 195.6 with perf(hip): wide stream rounds reuse cached weights, and a chunk's code pairs are made once #1694 too (195.2 to 196.0).
    • How often main lands slow varies from run to run. An earlier set of 6 servers an arm, with the first commit only, had 4 of 6 main servers slow: code 208.8 on main, 234.8 with the fix, 236.1 with perf(hip): wide stream rounds reuse cached weights, and a chunk's code pairs are made once #1694 alone, 240.2 with both. Its chat means were 188.1, 194.8, 197.6 and 197.0.

  • Qwen3.8-27B, --parallel 16, --levels 1,4,8,16 --reps 3, 2 servers a side: unchanged within noise.

    • Code at 1, 4, 8 and 16 streams: 13.3, 43.9-44.2, 47.7-47.8, 51.0-51.1 on main; 13.3, 43.9-44.0, 47.2-47.4, 51.0 with this PR.
    • Chat at 16 streams: 49.5-49.6 on both.
• Exactness:
  • Token hashes are equal on every server of every arm: 35B code 9f2923ac14d0, chat a8c28b8297b4; 27B code 2b28aa2108de, chat a06fab348fbe. No request failed.
  • tf-qwen35-test check: 27B --only invariants 42/0/5 and 35B-A3B 46/0/1, the same as main.
• Load time, the loaded in startup line:
  • 35B --parallel 4: 13.7 to 13.9 s on main, 11.0 to 11.4 s with this PR.
  • 27B --parallel 16: 35.5 and 35.9 s on main, 29.0 and 28.9 s with this PR.
• Prompt speed: n/a. The change only touches the load-time table that prices draft depth; the prompt path is untouched.
• Tests run:
  • New host tests in hip_costs_test.zig, built through the public hip_costs.forwards on a recording rig:
    • "every total row count prices at one forward of the row bucket it pads to, over the served streams" and "shared timings split a bucket's rows evenly over the served streams, past eight of them" both failed against the previous timing.
    • "a table past its entries still prices every total of the last bucket at that bucket's forward" fails when the last-entry rule is disabled.
    • "a total past the rows the served windows hold prices at the forward of the bucket it pads to" (4 streams, 128 batch rows: 65 rows price at the 4 × 32 forward, and every total from 1 to 128 at its bucket's forward) failed before the third commit with error.NotOneForward.
  • In hip_serve/costs.zig, "a forward of 8 rows reads steady in one run, so it runs once warm and once timed, and 7 rows keeps the default" failed before the second commit ("expected 1, found 3"). "a forward of long_rows rows or more runs once warm and once timed, a shorter one keeps the default runs" covers the threshold itself.
  • The previous behaviour still holds: "the lone window table stays one window of each width to the widest".
  • hip-host-test, test-qwen35-hip-host, check-native-hip-host and native-hip exit 0.
  • zig build test exits 0 on main ba7b2c4 and on this PR, both run from a git archive tree.
  • tools/zig/lean_check.py: 424 problems, the same as main. zig fmt --check is clean.
• Not run, and why:
  • Other HIP families: nemotron keeps its own forwards and is unchanged; I have no other HIP checkpoint locally.
  • gfx1030, gfx1100 and gfx1201: I have no card. The change is host-side timing, the same on every chip.
  • Metal and CUDA: untouched.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Tokens are unchanged: equal hashes on every server, and tf-qwen35-test check matches main.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (No kernel changes; the bits do not change.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. (No kernel changes.)
• Prompt processing is no slower than the last release. (The prompt path is untouched.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (gfx1151 only.)
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thanks @AbrahamCete. We read this through and ran the HIP host tests on it (hip-host-test, test-qwen35-hip-host and check-native-hip-host with -Dhip=true: 96 passed, 2 skipped). Tokens can't change: the cost table only feeds draft depth (core/lanes/depth.zig), and no kernel changed. Your receipts' equal hashes on every server agree.

One change before it lands:

• A bucket wider than streams × window is timed at the wrong size. hip_serve/costs.zig:78 caps the timed rows at streams * width, and :82 times @min(bucket, most). The table's last entry is then most, and core/lanes/allocate.zig:109-110 scales anything above it linearly. Your body's own example, 65 rows over 4 streams of 16, is this case. most is 64, and the round pads to the 128-row bucket (round_shape.zig:12-17). So 65 rows is priced at about 1.015 times the 64-row cost, not the 128-row forward's. Please time each bucket at its own rows, let the loop run to batch_rows, and add a test with 4 streams, width 16 and 128 batch rows that checks a 65-row total is priced at the 128-row forward. (The current tests all have most == batch_rows, so they never reach that path.)

A note, not blocking: one timed run per bucket instead of five makes each entry a single sample, so an unlucky sample can move the depth rule at a boundary. If the load-time budget allows, three timed runs for the shared buckets would lower that risk.

After the fix, the smallest check is one server per arm: Qwen3.6-35B-A3B MTP, --parallel 4 --context 4096 --prompt-cache-gib 0, then bench_concurrent.py --levels 4 --tokens 256 --reps 1 --temperatures 0, comparing your code and chat hashes (9f2923ac14d0, a8c28b8297b4). We don't have a gfx1151 here, so a receipt from yours would let us land it, with #1855 right behind.

[AbrahamCete]

Thanks @ashhart. Done in 493704a, and the branch is rebased onto main ba7b2c4 with no conflicts.

• buckets() now runs to batch_rows and times each bucket at its own rows, split evenly over min(streams, bucket) windows. Past streams × 16 the windows widen, so the 128-row bucket at 4 streams runs 4 × 32. round.shapeOf only bounds the total, so that forward has the same shape (128 rows, 8 slots) as a round padded to the bucket.
• width only fed the cap, so it left buckets(), forwards() and measure(), and hip_lanes.zig is back to main. The rig's scratch caches now hold the widest share: at + max(32, ceil(batch_rows / streams)).
• New test in hip_costs_test.zig, "a total past the rows the served windows hold prices at the forward of the bucket it pads to". It uses 4 streams and 128 batch rows: a 65-row total prices at the 4 × 32 forward, and every total from 1 to 128 prices at its bucket's forward over min(4, rows) windows. Before the fix it failed with error.NotOneForward, because the 65-row price came from the linear extension and not from a timed forward.
• One thing I found while checking it: the server sets batch_rows = streams * Hip.max_window (engine/native.zig:210), so most == batch_rows on every table it builds. The path could only be reached by calling buckets() directly, and no served table changes. The receipt agrees.

Receipt: one server per arm, alternated twice. Qwen3.6-35B-A3B MTP, --parallel 4 --context 4096 --prompt-cache-gib 0, then bench_concurrent.py --levels 4 --tokens 256 --reps 1 --temperatures 0:

arm	code tok/s	chat tok/s	code hash	chat hash	load
main ba7b2c4	216.5, 231.3	193.9, 193.5	9f2923ac14d0	a8c28b8297b4	13.4, 13.2 s
this PR 493704a	237.7, 237.8	192.6, 191.4	9f2923ac14d0	a8c28b8297b4	10.4, 10.4 s

hip-host-test, test-qwen35-hip-host and check-native-hip-host (-Dhip=true) exit 0. lean_check shows 424 problems, the same as main.

On three timed runs for the shared buckets, I tried a scratch patch (1 warm and 3 timed runs for shared forwards of 8 rows or more). Tokens stayed equal, and load time went up:

• 35B --parallel 4: 10.9 and 11.1 s, against 10.4 s with one run.
• 27B --parallel 16: 42.1 and 41.9 s, against 29.3 and 29.4 s with one run and 35.5 and 35.6 s on main. The 256-row bucket is most of that.

When I cut the runs, a single run from 8 rows up read within about 2% of the five-run mean on both checkpoints, so I kept one. If you'd rather pay the 27B load time for three runs, it's a one-line change and I'll add it.

[ashhart]

Thank you @AbrahamCete. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.