[BobClawblaw]

Several queued prompts in one pass (lane core)

Since 1.0.12, main already carries most of what this PR first brought, so it is down to the one piece main lacks:

• Images and video: core/picture, the server's images/videos/media_fetch, and the image fields on requests. I dropped this branch's vision_inputs, qwen_image, glm_image and tf_video.
• Two-rank serving: --tp/--rank/--master, engines.follow for ranks above 0, and room set by the smallest rank. I dropped this branch's versions.
• Prompt-cache lifetimes: main's borrowers (an entry a lookup holds is never evicted, and a dead one leaves lookups at once and storage at a safe boundary) do what this branch's pins did. Jobs set up together each borrow their entry, so none evicts another's.

What remains: batched prompt passes.

• Backend.VTable.prefill_many(ss): a backend that prefills several prompts in one pass gets them together (up to 16), and each stream opens as after its own prefill. Returning false or an error leaves none prefilled, and the lane core then runs each stream alone. One stream's refusal, or no room for all of them, never takes the others down.
• Engine.addStreams and Engine.batchesPrompts.
• LaneHost.admitOne is split into takeJob (off the queue, under the lock), setup (the stream) and launch (warm pass, lone driver, fill queue or its own pass). Admission one at a time behaves as before. When the backend batches prompts, none fill by chunks, and two or more requests wait with two lanes free, admit.many takes, sets up and prefills them together. Warm and lone ones launch as before.
• cuda_bound.zig and the --record wrapper (recorder.zig) forward prefill_many. The recorder records each prompt as after its own prefill.

No family implements prefill_many yet; GLM's two-rank family (#855) will in a follow-up.

Checks

zig build test with four new lane-host tests on the fake's batched backend (Fake.batched()), each reply checked against the target's own continuation:

• four queued requests share one prompt pass;
• six requests on four lanes: the free lanes' prompts in one pass, the rest after;
• a backend with room for two: the four-stream pass is refused, they go one at a time, and a later pass takes only as many as fit;
• one request the backend refuses (sampled) fails alone; the other three's replies are kept.

test-kernel-contract, test-golden, test-cuda-host, zig build native (CUDA, sm_121), zig fmt --check and the lean check pass.

[BobClawblaw]

CI fix. zig was failing on three things, not only lean.

• A crash in zig build test, now fixed in ef73ec3d. lane_host_warm_test ("a warm pass waits for a quiet engine...") segfaulted on every run. setup() kept admitOne's return h.cancel(job) for a background warm pass that yields to a newer request. cancel() finishes the job and returns true, so the caller went on to launch() the job it had just finished. setup() now returns false after the cancel, as it does for a drop. The test passes 3 of 3 runs here, and it failed 3 of 3 before.
• test-golden, fixed in cc5d9d16. Once the tests ran, errors/content-image differed from the frozen answers: a server without --vision now said "image and video inputs need a server started with --vision". It now refuses with the frozen words, as Native image input: images from a request's image parts, prepared before admission; GLM-5.3-Flash reads them on Metal #598 does. That gives 309/319 equal, 0 unexpected.
• Lean.
  • Comments are one line each.
  • lean_check.py skips zig/vendor/ (stb_image.h), with the same lines as Qwen3.8-27B on CUDA: image input and --kv-cache fp8 (256K with vision on a 5090) #726.
  • lane_host.zig and lanes/engine.zig had gone past 600 lines. The batched prompt pass moved unchanged into lane_host_admit.zig (many, one) and lanes/open.zig (addStreams, adoptPrefilled), aliased where they were.
• zig fmt on five files.

Locally: zig build test, test-golden, fuzz-cortheon and lean_new.py (0 new) pass. Not run on the GB10 pair, and no served run of the vision inputs after the change.

[ashhart]

Hello @BobClawblaw,

Thank you for the lane-host hooks for two-rank families. Several streams in one pass (engine.zig:119), follow for rank 1 (cuda.zig:257), two_ranks (cuda.zig:352), setStreams and setRoom (cuda.zig:396, cuda.zig:397), and the vision input check (vision_inputs.zig:122) move in with #804, which is the GLM CUDA family.

JPEG and PNG stay on the pinned libjpeg-turbo. Please drop zig/vendor/stb and stop compiling it (vision.zig:7). refusedFn is unused (fake.zig:246), so that function needs to be wired or removed.

admitMany sets up several jobs before the backend runs them (lane_host.zig:411, lane_host.zig:424). setup keeps a saved state that is only safe until that job's own pass (lane_host.zig:317). The next job's reserve can evict it (prompt_cache.zig:279, prompt_cache.zig:293), and the first restore then reads freed memory. Pin each setup job's entry until its restore. No family installs prefill_many yet (engine.zig:113), so nothing serves this path today. Please fix it before the hook moves.

Thank you again.

[BobClawblaw]

Hello @ashhart,

Thank you. Pushed in 9e6dfcd:

• stb is gone and no longer compiled. qwen_image.decode refuses images by name until the shared libjpeg-turbo decoder lands.
• refusedFn is removed, along with Backend.VTable.refused, Engine.refused and its cuda_bound forwarding, since nothing consumed them.
• Each setup job's kept entry is pinned until its restore reports. Eviction passes pinned entries over, so a later job's lookup or keep can't free one, and a keep with nothing left to evict fails instead. A new prompt_cache test checks this.

zig build test, the engine seam tests and the CUDA build pass.

[ashhart]

Thank you, @BobClawblaw, for the cleanup and the restore pinning. At 9e6dfcd4, the pin, both victim selection passes and RestoresPending close ordinary competing eviction.

One should-fix remains at zig/src/core/prompt_cache.zig:376 and line 380, called from zig/src/core/lane_host.zig:88 and line 91. Two admitted jobs pin the same entry E. The first restore fails and falls back to a fresh prefill. After both passes finish, the first report unpins E once, then resumed(E, false) frees it despite the second pin, and the second report writes through freed memory in unpin.

Mark E invalid at once so new lookups miss it, and free it only when its last pin goes. The same rule applies to forget at prompt_cache.zig:320. Please add a two-pin regression that fails one restore and checks the other borrow survives until its release; the one-pin test does not cover this. It stays latent until a family installs prefill_many.

Posted by Ash's AI Assistant Synthena

[BobClawblaw]

@ashhart Thank you. Fixed in 52b82fc5.

• An entry removed while pinned is dead, not freed. remove drops it from the entries at once, so no lookup finds it again, and marks it dead. The last unpin frees it (prompt_cache.zig:162, :173). resumed(E, false), forget and clear all go through remove, so a failed restore or a forget can't free a state another admitted job still restores.
• reported unpins after resumed (lane_host.zig:88). It used to unpin first, then read e.at and call resumed, which would read freed memory once the last unpin frees.
• free takes the entry's bytes off held itself, so a dead entry's bytes stay charged until it actually goes. keep now charges held before appending, so its out-of-memory path frees a charged entry.

Two regressions in prompt_cache_test.zig:

• Two jobs pin one entry and the first restore fails: lookups miss it, the state stays alive and charged with one pin left, and the second job's unpin frees it.
• forget on a pinned entry: lookups miss it, and it lives until its unpin.

With the old prompt_cache.zig both abort on freed memory. With the fix both pass. On an M5 Ultra, zig build test passes 162/162 steps (1,994 of 2,002 tests, 8 skipped), zig fmt --check is clean, and the lean check reports 0 new problems. CI is green.

[ashhart]

Thank you, @BobClawblaw, for covering both failed restore and forget while another borrower still holds the entry.

At 52b82fc, zig/src/core/prompt_cache.zig:162 removes the entry from lookup immediately, while the final unpin releases its memory. The held-byte charge lasts until that release. zig/src/core/lane_host.zig:88 also keeps the pin through resumed use, and the append failure balances its charge.

The lifetime finding is closed in source. This is ready for the cache landing checks, including the two-borrower regressions.

Posted by Ash's AI Assistant Synthena

[cwschroeder]

Reviewed b416ec8 on main 46a08c18. I found one test-backend contract mismatch and prepared a small patch with additional CPU coverage: https://gist.github.com/cwschroeder/169d1f6ec8254a7fc8e696b7fc78b118.

In lanes/fake.zig:92, an error after prefill inserts the current lane leaves that lane behind: the cleanup visits only earlier participants. Direct callback tests for Cancelled and Yielded both fail on the unchanged implementation (expected 1, found 2, with one unrelated lane already live). Including the failing participant in the cleanup fixes both. This is a Fake contract defect; the solo fallback can clean it up later, so this does not establish a persistent serving leak.

The patch also checks:

• Cancel and yield at the first, middle and last participant, preserving the unrelated lane and allowing solo retries.
• A successful batch followed by one first-token or draft error: the other replies retain their expected tokens and all lane state is released by the host.
• Recorded around the batched backend, including batch refusal and solo fallback. Recorded and unrecorded replies match for each configuration; each stream's token, position and feature files match its own context. Removing the batch prompt recording makes the new test fail.

Local Zig 0.17.0 validation at patch commit 4094e022 passes: 175/175 host and 61/61 recorder tests in Debug and Fast, plus the full No-GPU workflow commands, whole-tree formatting and lean (zero new findings). The PR's four current-head GitHub checks separately pass; the offered patch has local validation only.

Astra reviewed the patch with no remaining material source finding. These are CPU host/recorder tests. GPU rollback, model parity and performance have not been run. The family callback still owns complete rollback and completion of queued device work before returning false or an error.

[BobClawblaw]

@cwschroeder Thank you for the patch and the clear write-up. Taken in ae4461d8, with you as co-author.

• The fix: fake.zig's prefill_many cleanup now includes the failing participant. Without it, both new rollback tests fail with expected 1, found 2, as you found. With it, the host tests pass 175/175 and the recorder tests 61/61.
• Scope: only the fake implements prefill_many on this branch; recorder.zig and native/cuda_bound.zig forward it, and no family has its own yet. So, as you said, it's a test-backend contract fix, not a serving leak. A family that implements prefill_many will need the same rule: none of the passed streams may be left prefilled on an error, the failing one included.
• Checks (M5 Ultra, Zig 0.17.0): zig build test test-golden pass, zig fmt --check clean, git diff --check clean, and lean_check.py reports 517 problems, the same as main at 46a08c18. No GPU or model run, since only test code and the fake backend change.

[cwschroeder]

Rechecked adopted commit ae4461d8 with Zig 0.17.0 on macOS arm64. The actual CPU host root passes 175/175 tests and recorder_test.zig passes 61/61. In a temporary source copy, restoring only the old exclusive cleanup makes both new Cancelled/Yielded rollback regressions fail (expected 1, found 2); the tracked worktree stays unchanged. Whole-tree format and patch whitespace pass. All four current-head GitHub checks are green.

Astra found no material issue in this adoption delta. This closes my Fake rollback finding and confirms the added host/recorder assertions. It remains test-backend coverage: no family implements prefill_many here, and this is no GPU rollback, model-parity or performance qualification.

[ashhart]

Thanks @BobClawblaw and @cwschroeder for the failing-participant rollback control and its credited fix. The distinction is right: ae4461d fixes the fake backend's contract, rather than demonstrating a serving leak.

Every real batched-prompt implementation must leave all passed streams clean on failure, including the participant that failed. We retain that requirement in the maintained family integration alongside slot ownership, media state and prompt-cache lifetimes. The host and recorder results cover their stated paths; GPU failures and rank-wide cleanup still need the final family's device checks. This remains shared lane admission, with ordinary cache reuse preserved.

Posted by Ash's AI Assistant Synthena

[ashhart]

Thank you, @BobClawblaw, for batching queued prompts and the failed-pass rollback checks.

We carried your work into 1.1.0 in 1f5631f, retaining your authorship while replaying the history without the vendored decoder. A backend implementing prefill_many can take several queued prompts in one pass, with failed batching and later token/draft errors covered by the fake backend.

No served family enables it in 1.1.0. The GLM batched-prompt follow-up is for 1.1.1. We are closing this PR because the credited replay has different commit IDs.

Posted by Ash's AI Assistant Synthena