perf(metadata): scrub the padded input tail in one launch - #1218
Open
nperrin-fr wants to merge 3 commits into
Open
perf(metadata): scrub the padded input tail in one launch#1218nperrin-fr wants to merge 3 commits into
nperrin-fr wants to merge 3 commits into
Conversation
Every step refreshes the padding tail of six persistent graph input buffers, because a larger prior iteration leaves stale token ids, cache locations and request-pool slots where the captured graph still reads. As six torch fills that is six launches on the pre-replay critical path for a few kilobytes; in a profiled decode the six are the largest remaining block of elementwise work there, and none of it overlaps other streams. PrepTape already exists for exactly this and already spells the operation (filltail), so the scrub becomes one recorded tape replayed per step. It is recorded against the persistent buffers in the constructor, so no model supplies a kernel of its own; every model's input prep gets it. Three of the six buffers index a request pool and are int64, which the tape could not address, so filltail now carries its destination width in the descriptor's spare stride field and the interpreter stores through a 64-bit pointer for those rows. Mixed widths coexist in one stage. A non-CUDA InputBuffers keeps the torch spelling, which the backends' CPU tests need. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DxaFPVQF6L13TGgrbwnDdh Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c044aeabc7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
nperrin-fr
marked this pull request as draft
August 23, 2026 17:13
PrepTape wrote each step's register values straight into one persistent pinned buffer and uploaded it asynchronously. Callers dispatch step N+1 before step N commits (event_loop's in-flight depth), so the CPU can reach the next step's write while the previous upload is still queued, and the in-flight tape then reads the newer registers -- a row count that no longer matches its buffers, or a pointer slot naming a table its own step never owned. input_buffer's bulk pinned staging avoids exactly this and says why. Values now live in a plain host tensor and each run copies them into a freshly allocated pinned block, which the caching host allocator does not hand back until that copy's stream event has completed. Measured on GB200, a run costs 46.8us of dispatch against 38.0us before, still well under the 67.4us of the six torch fills it replaces on the input path. Rewriting the pinned buffer under a queued upload does change what the tape reads -- a probe that mutates it between run() and the sync flips the filled range. Two ordinary consecutive run() calls did not reproduce that, so this is a hazard closed by construction rather than a reproduced failure, and it comes with no test that fails without it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DxaFPVQF6L13TGgrbwnDdh Signed-off-by: Nicolas Perrin <nperrin.ai@gmail.com>
nperrin-fr
marked this pull request as ready for review
August 24, 2026 07:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every step refreshes the padding tail of six persistent graph input buffers, because a larger prior iteration leaves stale token ids, cache locations and request-pool slots where the captured graph still reads. As six torch fills that is six launches on the pre-replay critical path for a few kilobytes; in a profiled decode the six are the largest remaining block of elementwise work there, and none of it overlaps other streams.
PrepTape already exists for exactly this and already spells the operation (filltail), so the scrub becomes one recorded tape replayed per step. It is recorded against the persistent buffers in the constructor, so no model supplies a kernel of its own; every model's input prep gets it.
Three of the six buffers index a request pool and are int64, which the tape could not address, so filltail now carries its destination width in the descriptor's spare stride field and the interpreter stores through a 64-bit pointer for those rows. Mixed widths coexist in one stage. A non-CUDA InputBuffers keeps the torch spelling, which the backends' CPU tests need.
Summary
Test Plan