Repository navigation
CUDA: a request's stats list its prompt pass widths for the 27B, GLM and Nemotron (stacked on #2293) - #2304
Open
ashhart wants to merge 4 commits into
Open
CUDA: a request's stats list its prompt pass widths for the 27B, GLM and Nemotron (stacked on #2293)#2304ashhart wants to merge 4 commits into
ashhart wants to merge 4 commits into
Conversation
Count the actual target slices in the shared Metal prompt path. Keep verify and drafter absorb work out of the list and preserve inference when observation allocation fails. Forward the request-owned list through the lane host and existing raw/chat runtime fields. Host regressions cover stepped and serial cuts, the lone scheduler, drafted verify exclusion, observation failure, mailbox ownership, and whole/streamed raw and chat replies. Refs #2089, #2120.
Share the request-owned observation policy with Qwen27. Count GLM chunks after completion and Nemotron prompt buffers at their existing landing wait, preserving its one-chunk lookahead. Clear abandoned observation owners on a failed release drain. Host cases exercise chunked and one-pass prompts, actual cut planning, completion and ownership, verify exclusion, and request stats. Add shader-free compiles of the real family callbacks. Refs #2089, #2120.
Owner
Author
|
CUDA cells for this change, main against this branch:
The Metal side was checked with #2293. This lands right after #2293, which it's stacked on. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2293, whose three commits show here until it lands. This PR's own change is the last commit.
On CUDA, a request's finished stats now list
tensorfold.prefill_widthstoo, for the three CUDA families we run: Qwen3.8-27B (and Qwen 3.5), GLM-5.3-Flash and Nemotron. Each entry is one completed target prompt pass with the rows it ran. An entry is added only where the code already waits for that pass to finish; no wait, sync, loop or cut is added or moved..lastforward, whose readback synchronizessettle_stepwaitprefillManyreturns, since it draws each stream's tokenprefillFrom's existing first-token download, only when exactly one target chunk ranglm/cuda_lanes.zigandnemotron/cuda_lanes.zigwere already at 600 lines, so each drops seven separator blank lines to stay at the limit. Each also exposes an existing step to the host fixture. Telemetry only: no token, default, kernel or launch change.Checks:
test-cuda-host test-runtime-host check-native-cuda-host test-lane-api-hostand the widths steps.zig build -Dtarget=aarch64-linux-gnu compile-cuda-tests: 10 of 10 steps.zig fmt --check; lean is unchanged.