Skip to content

Metal: a request's stats list its prompt pass widths, on every text family - #2293

Merged
ashhart merged 4 commits into
mainfrom
lead/prefill-widths
Nov 6, 2026
Merged

ashhart merged 4 commits into
mainfrom
lead/prefill-widths

Conversation

@ashhart

@ashhart ashhart commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

A request's finished stats now say how its prompt was cut on Metal: tensorfold.prefill_widths lists one entry per completed target prompt pass, with its real row count, for Qwen3.8-27B, Nemotron, GLM-5.3-Flash and every other Metal text family: Gemma 4, LFM2-MoE, Step5, Spark2.5, GLM-DSA, Kimi K3, Llama, MiniMax-M3, Qwen 3.5 and Qwen4. It answers "how was my prompt chunked?" without a profile build.

  • What it counts: only completed target prompt passes. A drafted verify round isn't a prompt pass, and neither is the drafter absorbing the prompt. A pass that fails is dropped. If an allocation fails, the stats come back without widths and the reply still succeeds.
  • Where it's counted:
    • The 27B appends in the shared Metal fill path.
    • GLM appends after its synchronous prompt chunk.
    • Nemotron appends when each pipelined prompt command buffer lands (moved into prompt_flight.zig, with the same chunk cuts, wait, error and timing path).
    • The other ten families each get one added line at the point their existing prompt loop finishes a target pass. No loop is moved, and no original line changes.
  • Telemetry only: no token, default or kernel changes. The only additions to the hot path are a list append, and a field on the lane stream that is freed with it.

Checks (the third commit adds check-metal-prefill-rest-host and test-qwen4-prefill-widths-host: 31 of 31 steps and 16 of 16 tests, with the earlier steps):

  • zig build test-qwen27-prefill-stats-host test-qwen27-prefill-widths-http-host test-glm-prefill-widths-host test-glm-scheduling-host test-nemotron-prefill-widths-host check-nemotron-prefill-widths-host: 19 of 19 steps, 78 of 78 tests. Chunked prompts report their real cuts, a one-pass prompt reports one entry, a resumed prompt reports only what it ran, and a failed chunk reports nothing.
  • zig fmt --check on every touched file; lean unchanged at 426.
  • A plain zig build passes: 255 of 255 steps.

@ashhart

ashhart commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner Author

The LFM2 rerun on an Apple M5 Ultra is level (LFM2.5-24B-A2B 4-bit, main against this branch, warm base new new base base new, three runs an arm):

  • Prefill is within 0.4% at 16K, 32K and 64K, at every start offset.
  • Decode medians are within 1.6% at every cell but one.
  • The exception is -3.3% at 16K from offset 5,000. There the branch's three runs are 277.9, 264.3 and 263.9 tok/s against main's 273.3, 268.2 and 274.6. The runs overlap, with the branch's fastest run faster than any of main's, so it isn't a steady change.

With the first LFM2 run's REVIEW now settled, this lands from the queue, and #2304 follows once its CUDA cells are in.

Count the actual target slices in the shared Metal prompt path. Keep verify and drafter absorb work out of the list and preserve inference when observation allocation fails. Forward the request-owned list through the lane host and existing raw/chat runtime fields.

Host regressions cover stepped and serial cuts, the lone scheduler, drafted verify exclusion, observation failure, mailbox ownership, and whole/streamed raw and chat replies. Refs #2089, #2120.
Share the request-owned observation policy with Qwen27. Count GLM chunks after completion and Nemotron prompt buffers at their existing landing wait, preserving its one-chunk lookahead. Clear abandoned observation owners on a failed release drain.

Host cases exercise chunked and one-pass prompts, actual cut planning, completion and ownership, verify exclusion, and request stats. Add shader-free compiles of the real family callbacks. Refs #2089, #2120.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant