Skip to content

CUDA: a request's stats list its prompt pass widths for the 27B, GLM and Nemotron (stacked on #2293) - #2304

Open
ashhart wants to merge 4 commits into
mainfrom
lead/prefill-widths-cuda
Open

ashhart wants to merge 4 commits into
mainfrom
lead/prefill-widths-cuda

Conversation

@ashhart

@ashhart ashhart commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Stacked on #2293, whose three commits show here until it lands. This PR's own change is the last commit.

On CUDA, a request's finished stats now list tensorfold.prefill_widths too, for the three CUDA families we run: Qwen3.8-27B (and Qwen 3.5), GLM-5.3-Flash and Nemotron. Each entry is one completed target prompt pass with the rows it ran. An entry is added only where the code already waits for that pass to finish; no wait, sync, loop or cut is added or moved.

family where it appends what it records what it leaves out
Qwen 3.5 / 27B CUDA after the chunk's existing .last forward, whose readback synchronizes every target chunk's real rows, including request cuts and resumed tails —
GLM stepped after the final draw, or after an existing intermediate settle_step wait the rows each step took intermediate steps with no settle wait
GLM batched after prefillMany returns, since it draws each stream's token each request's prompt length, once —
Nemotron served after prefillFrom's existing first-token download, only when exactly one target chunk ran that chunk's rows multi-chunk calls, which report nothing rather than guess

glm/cuda_lanes.zig and nemotron/cuda_lanes.zig were already at 600 lines, so each drops seven separator blank lines to stay at the limit. Each also exposes an existing step to the host fixture. Telemetry only: no token, default, kernel or launch change.

Checks:

  • Host: 339 of 339 steps, 5,013 tests passed and 28 skipped, across test-cuda-host test-runtime-host check-native-cuda-host test-lane-api-host and the widths steps.
    • Chunked, resumed and one-pass prompts report their real cuts in all three families.
    • With the five appends removed, five of the ten width tests fail.
  • zig build -Dtarget=aarch64-linux-gnu compile-cuda-tests: 10 of 10 steps.
  • zig fmt --check; lean is unchanged.

Count the actual target slices in the shared Metal prompt path. Keep verify and drafter absorb work out of the list and preserve inference when observation allocation fails. Forward the request-owned list through the lane host and existing raw/chat runtime fields.

Host regressions cover stepped and serial cuts, the lone scheduler, drafted verify exclusion, observation failure, mailbox ownership, and whole/streamed raw and chat replies. Refs #2089, #2120.
Share the request-owned observation policy with Qwen27. Count GLM chunks after completion and Nemotron prompt buffers at their existing landing wait, preserving its one-chunk lookahead. Clear abandoned observation owners on a failed release drain.

Host cases exercise chunked and one-pass prompts, actual cut planning, completion and ownership, verify exclusion, and request stats. Add shader-free compiles of the real family callbacks. Refs #2089, #2120.
@ashhart

ashhart commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner Author

CUDA cells for this change, main against this branch:

  • Qwen3.8-27B on a GB10: GREEN.

  • Nemotron on a GB10: GREEN.

  • GLM-5.3-Flash NVFP4 at TP=2 on two GB10s: tokens equal in all 52 checked cells (0 unequal, 0 unverified). Speed, from two runs an arm:

    • prefill is 6.7% faster at 2K and 4.5% faster at 8K and 32K;
    • decode is +1.2% (chat) and -1.6% (code) at one stream, and +0.3% (chat) and +5.4% (code) at two.

    The prefill gain would need more runs before we'd claim it. The -1.6% is inside the spread we usually see with two runs an arm.

The Metal side was checked with #2293. This lands right after #2293, which it's stacked on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant