lanes: a family's round planner drives the lane host with prompt pieces, and a halt ends the rounds at a drain deadline (DeepSeek-V4.1 stack 6/14) by jayleaton · Pull Request #662 · ashhart/TensorFold

jayleaton · 2026-10-11T11:29:06Z

What this changes

lanes: a family's round planner drives the lane host with prompt pieces, and a halt ends the rounds at a drain deadline. Depends on #637. This cumulative PR targets main; the incremental range is 0ba7d305..e9f9210b.

#637 → #662 → later DeepSeek-V4.1 layers

Commits in this layer:

09188093 Let a family's round planner drive the lane host with prompt pieces
361f0634 Lane host halt: a stop's drain deadline ends the rounds at the next boundary, and every request queued, filling or running fails with the reason.
e9f9210b Round planner test: both prompts queue before the host starts, and its waits time out

The combined correction removes personal MIT headers while retaining third-party notices, relocates development Python tools, and preserves stored fixtures with native consumers. Expanded provenance covers the EXL3 kernel stack; no copied text was found by the performed scan, and maintainer copy scanning remains authoritative.

Receipt

Before: public head 006908c0. After: exact corrected head e9f9210be2ed986edc61cbe57e5d431570215ce4, gated on 2026-10-12, Linux x86_64 and Zig 0.17.0.

Executed gate output:

G662: commit e9f9210be2ed986edc61cbe57e5d431570215ce4 tree 62a562b56103d2994dd1d58d8be428c045023a77 dirty 0 2026-10-12T04:42:42Z
install rc=0 Build Summary: 12/12 steps succeeded
test rc=0 Build Summary: 44/44 steps succeeded; 542/543 tests passed (1 skipped)
test-tp rc=0 Build Summary: 4/4 steps succeeded
zig fmt --check rc=0
git diff --check rc=0
e9f9210b lean: 1226 files, 534 problems | new: 0 | gone: 0
FAILED STEPS 0

Lean uses pinned main e7c523eb: 534 inherited findings, zero new or removed identities. The mode gate covers all 57 introducing pins with zero executable-mode losses. Family suites and CLI receipts run where introduced. Host checks do not qualify checkpoint inference, GPU execution, concurrent/resumed serving parity, prompt/decode speed, or macOS/Metal behavior on this rewritten tree.

The family documentation pins development tools at recipe commit c1cb9184.

Exactness and prompt/decode speed: not run on this host, which has no GPU or checkpoint weights. No new performance claim. MLX/PyTorch/checkpoint revision: not applicable to the host gate.

Checklist

Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
No precision traded for speed. If the bits change, the description says which sums change and why.
Prompt processing is no slower than the last release.
The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
New tests fail before the change, pass after it, and skip cleanly without their dependency.
Comments and docstrings are one line. No measurements or history in the source.
No personal data, machine names, internal hosts or local paths. No AI attribution lines.

Merge Danger

Door: two-way. Reverting restores the parent without a data migration.

Blast Radius: engine. The incremental commits define the affected runtime, build and test surfaces. The production branch is unchanged.

…alls them. graph_cache.Cache captures one executable graph per caller-named key and replays it when the key comes back; ranks agree on every capture, eviction and fallback through an Agree hook, and a memory floor turns misses eager. arena.Arena sub-allocates DeviceBuffer.alloc from a few large chunks (VMM or plain cuMemAlloc) through memory.setHook; borrowed spans and pointers it does not own are freed as before.

…; CUresult names fall back to a table when the driver cannot answer. cuFuncGetName is optional (CUDA 12.3+), so an older driver still opens. During the exit's teardown cuGetErrorName fails too, which printed CUDA_ERROR_DEINITIALIZED as "CUDA_ERROR_UNKNOWN_CODE (4)"; code 4 now says the process is exiting.

Use CUDA device attribute 110 for GPUDirect RDMA with VMM support. Attribute 126 reports the memory sync domain count and can incorrectly enable the RDMA allocation flag. Add a regression for the external ABI contract.

…r stays under the max, a max of 0 is eager - A capture whose agreement throws, or whose insert runs out of memory, no longer leaks its executable: the map's slot is reserved before the capture, and the executable is owned here until the insert, which cannot fail. - A host map that cannot grow counts as no room in the existing room vote, so every rank declines that miss together. - `min_keep` is clamped to `max`, so under the memory floor the cap never rises above the configured max. A max of 0 runs every step eagerly, with no collective. - A cap lowered to 0 under the floor goes back to max once every rank has room, instead of staying eager for good. - Hits make no collective: with `Agree` set, keys, settings and fingerprint changes must already match on every rank. The docs now say so: the fingerprint changes at the same step on every rank, and `fingerprintOf` is for one rank. Co-authored-by: Jonathan Regan-Beasley <197533392+jregan-beasley-smc@users.noreply.github.com>

The arena's `live` (small ranges) and `large` (own chunks) maps drop an entry on every free and gain one on every allocation for the life of the process. std's tombstones never clear, so in time they take every free slot, and each allocation, free and miss then probes the whole map. Each map now counts its removals and is rehashed in place once they reach a tenth of its capacity. Test: 20,000 random allocations and frees (small ranges and large chunks) keep a tenth of each map free at every step; without the rehash it fails.

…d RoCE collectives, plan link, fail-fast. zig/src/fabric/tp brings a rank up from TF_TP_WORLD, TF_TP_RANK, TF_TP_MASTER and TF_TP_PORT, agrees the settings digest over every rank, and gives a family all-gather, all-reduce and broadcast over NCCL (TF_COMM_BACKEND=nccl, the default), a pinned host mailbox for ranks on one host, or RoCE verbs between two hosts (TF_TP_ROCE_*), with an NCCL fallback when RoCE setup fails. Rank 0 sends each round's plan to the others over its own link (TF_TP_PLAN_*). A failure or a vanished rank ends every rank shortly after (TF_TP_FAILFAST, TF_TP_FAILFAST_GRACE_S). No family uses it yet. zig build test-tp runs the host tests; zig build tf-tp-test builds the two-GPU runner (-Dtp-nvcc= or -Dtp-fatbin= for the mailbox kernel). The RoCE all-gather adapts b12x's RoCEnante protocol.

… the stop word with no fail-fast exit; a rank killed mid-drain still fails the other fast.

… /24 the peer also has The automatic pick took the first N active RoCE ports in device-list order and never checked that the peer can reach them. On a host with two neighbours, one picked port can sit on a subnet the peer does not have, and the queue pair's RTR step then fails, which reports RoCE unavailable. Every rank now sends its usable ports' IPv4 addresses over the bootstrap. Each rank then keeps the ports whose /24 the peer also lists, ordered by subnet and each /24 no more often than both sides have it, so both ranks pick the same number. With no shared /24 the old first-N order is kept, with a warning to set TF_TP_ROCE_HCA. The exchange runs on every rank, including ranks that set TF_TP_ROCE_HCA, so the bootstrap stays in step.

…FNAME and TF_TP_MASTER are needed, with a two-host example

… kernel backends up front; run_hosts.sh passes HCA picks - `tf-tp-test kernel` reports the embedded kernel and exits 1 without one. `suite` and `failfast` on the mailbox or RoCE backend stop at once with a message that names `-Dtp-nvcc`, instead of failing later on a missing kernel. - `run_hosts.sh` stops before any step when the kernel is missing. It exports TF_TP_ROCE_HCA and NCCL_IB_HCA (derived from TF_TP_ROCE_HCA when unset), warns when a host has more than one RDMA device and neither is set, and records all three variables in host.txt.

… with another region layout are refused roce_guard.stripes computes each HCA's share of a post (its send stripe, the peer's receive range and its flag word) and refuses with error.OutOfRegion any range past its slot, its flag area or the registered region, before post_send. The verbs wire and the shm wire both post through it. The shm wire moves to roce_shm.zig and can stripe like the verbs wire (Options.shm_stripes), with an optional tap that sees every post for tests. The connection record carries the slot and registered length, so a peer with another layout is refused at connect; its magic is bumped. Tests: stripes tile the slot over two HCAs and refuse a post past the slot, a third slot, an unpadded or empty post, no HCA and a short region. On fake verbs a post past its slot reaches no verb, and a full slot stays inside the region. Over the shm wire striped as two HCAs, 2,000 variable-length gathers stay inside slot and region, bit for bit.

…would ask for one A post of about 9.6 MB faulted a DGX Spark's HCA. roce_guard.max_write (1 MiB, a constant) is checked in stripes() before the slot bounds, so the verbs wire, the shm wire and the proxy refuse a larger post with error.WriteTooLarge before any verb, whatever the slot size. Roce.init refuses a slot past it. Config.parse fails boot with a clear message when TF_TP_MAILBOX_MAX_KB would make the RoCE slot pass 1 MiB; the value is never clamped. Tests: a 9.57 MB and a 1 MiB + 16 post are refused in a 10 MiB slot and reach no verb, a 1 MiB post goes out; boot refuses TF_TP_MAILBOX_MAX_KB 1028, 2048 and 16384 and keeps 1024.

…sized ops, at most 2 MiB in flight Default off. allGatherV strides past the slot, up to TF_TP_ROCE_CHUNKED_KB, go RoCE as ceil(n / slot) launches of the tp_roce kernel: op c moves at most a slot of each shard from c x slot (RoceArgs.chunk), and an op empty on both ranks rings no doorbell and leaves the epoch. Every RDMA write keeps the 1 MiB cap. The kernel poisons the transport rather than stage past a slot, and its four failure-record blocks fold into fail_record (same stores and fences). VerbsWire.post waits for completions while the bytes posted and not completed over both HCAs would pass max_inflight (2 MiB, a constant in the post path); a wait that sees no completion for 10 s fails the proxy. Boot refuses TF_TP_ROCE_CHUNKED_KB when the kernel image lacks the tp_roce_chunk_abi marker. The setting is part of the digest both ranks agree on. Tests: chunked strides from 300 KB to a 128 MiB union posted back to back on fake verbs whose completions lag keep at most 2 MiB in flight and no write past 1 MiB, and reach the cap; the wire's own in-flight count and peak match the fake HCAs'. Over the shm wire, 2,000 chunked variable-length gathers of up to four slots arrive bit for bit.

…region; the region is DONTFORK and NOHUGEPAGE A local protection error can arrive while the IOMMU invalidates its mappings lazily. On such an error the old path left the other HCA's QP ready to send (still taking the peer's writes), hung up the fate channel without telling the peer to stop, and freed the region at once: unregistered from CUDA with no GPU sync, then QP destroy, MR deregistration and munmap. A device write after the unmap could then reach a reused page through a stale IOTLB entry. - Roce.fail and Roce.abort (the fail-fast hook) move every QP to the error state: no send goes out and no peer write lands. - Roce.deinit quiesces first: QPs to error, every posted request reaped (flushed or not), cuCtxSynchronize. A failed transport, or one that does not quiesce, keeps its region registered and mapped; the process exit tears it down. - Session.close on a failed transport goes fail-fast before closing, so the peer aborts and halts its HCAs first. - The region is MADV_DONTFORK and MADV_NOHUGEPAGE before ibv_reg_mr; a refusal fails the start. - A QP whose actual max_inline_data is under 4 B is refused (the flag post uses lkey 0 on a stack word). - One log line a rank: region base and length, each HCA's lkey and rkey, the peer's region and rkeys. Tests: on fake verbs Roce.abort halts both QPs; quiesce halts both before the first flush and reaps every posted write; an aborted transport reports itself failed, and over the shm wire the aborted rank keeps its region while the other frees it; smaps shows dc and nh on a prepared region; the inline bound refuses 3 B and takes 4.

…path on real HCAs, only on an idle link - `tf-tp-test failfast V transport [N [HOLD_S]]`: N clean 4 KiB sums, the ranks meet over NCCL, each waits for its link to go idle (every HCA's posts completed), both agree over NCCL, then the victim takes the proxy's failure path (`Roce.inject`, which is `Roce.fail`) and closes its session: both ranks halt their QPs and exit 70, and the victim's region stays registered. When either rank's link is not idle there is no inject, a clean stop, REFUSED and rc 2. - Its host-only forms (TF_TP_FAILFAST=0, or a hold) are refused unless TF_TP_ALLOW_HOST_ONLY=1 and no RDMA wire is in use (the shm wire, or no RoCE), so they never run on an HCA. - `run_transport.sh MASTER PEER_SSH OUT`, driven from rank 0's host (rank 1 over ssh): prechecks on both hosts (no GPU process, memlock unlimited), then T0 (suite), T1 and T3 (each rank as the victim). After each step it reads both hosts' kernel log since the step and every HCA *err* counter; a hit, a counter that grew, an unexpected rc or a failure line prints STOP: and exits 1. `run_transport.sh t2 OUT` runs the host-only form on one host. - The `halted` and `quiesced` lines carry each QP's state read back with ibv_query_qp (an optional symbol), each HCA's posts and completions, the completions the drain reaped and how many of them were flushes. - `Roce.allGatherV` refuses a stride past the slot (one op; the route never sends one). - The README states the RoCE limits and how to run the check. Tests: on fake verbs the link reads busy with two posts out and idle after the drain, and the report line names both QPs in the error state with every completion a flush.

The prefix index keeps one node per distinct full page of every saved entry in `by_digest`. Each saved turn adds its new pages, and each entry dropped or evicted removes the pages no other entry passes through, for the life of the server. std's tombstones never clear, so in time they take every free slot, and each save's new pages and each find's walk past the cached prefix then probe the whole map. The map now counts its removals and is rehashed in place once they reach a tenth of its capacity. Test: 3,000 turns of up to 60 fresh pages, the oldest dropped past eight, keep a tenth of the map free after every turn; without the rehash it fails.

A family can now hand the lane host a round planner (api.Rounds) that admits requests, splits prompts into pieces and runs them in the same rounds as the other streams' decode rows. The planner is opt-in: the host's own loop, with a backend's prefill_step a chunk a loop between rounds, stays the default, and a host with a planner runs the planner's loop instead (LaneHost.run picks one). The backend gains optional entries for this: begin, piece, pieces (several streams' pieces in one call), finish and join, so a round's finished prompts get their first drafts in one backend call. A planner's prompts open through lanes/open.zig like the whole and the chunked pass (open.openedWith; their first drafts sized by the head's reach under node probabilities), and a prompt cancelled at its final releases its lane as a cancelled chunk does. The lane core also takes a family row policy (lanes/trim.zig) in place of the built-in allocation, proposers that price their own drafts, and a timed switch for backends without round timers. Under a planner the host still steps the learner while no request waits or runs, and a failed piece or piece run fails only its own requests. A planner over a backend without prompt pieces (such as the external drafter wrapper) is refused at start with error.PiecesUnsupported. The round loop lives in lane_rounds.zig, a request's stream setup in lane_stream.zig and the pieces' steps in lanes/pieces.zig, keeping the host and the engine under the line limit.

…oundary, and every request queued, filling or running fails with the reason. Opened.halt (optional) lets a server stop an engine between rounds instead of under a running one; LaneHost.halt is that hook for lane-host families. A request still filling its prompt a chunk a round fails too, with its lane freed, and a prompt pass the halt cancels mid-chunk reports the halt's reason rather than a cancel.

…s waits time out The second prompt was submitted after start(), so a loaded scheduler could let the first finish its pieces before the second arrived: no queued admission, and the test's waits > 0 check failed. Both are queued now before the first round. Each wait gives up after 10 s, and a timed-out wait stops the host thread before the test's frame is gone.

… TF_TP_ROCE_HCA nor NCCL_IB_HCA is set - `tf-tp-test hca` (host only, no GPU): both ranks run the RoCE pick over the bootstrap, the same rule the wire uses, and each prints its own devices as `NCCL_IB_HCA=name,...` (no GIDs). It exits 1 with the reason when the host has no usable port or no port on a /24 the peer shares, so no rank falls back to its first ports for NCCL. - run_hosts.sh runs it when neither variable is set, exports the names for every step and logs each rank's pick; a failed or malformed `hca` stops the runner (exit 3) with the reason; `hca` is capped at TF_TP_PLAN_TIMEOUT + 31 s, so its own reason (no peer joined in time, port in use, the peer closed, a bootstrap error) comes first. An explicit NCCL_IB_HCA, an explicit TF_TP_ROCE_HCA (NCCL_IB_HCA derived from it) or both keep their behaviour; the many-devices warning is gone. Both hosts use the same mode: with one naming its ports and the other not, the latter's `hca` finds no peer and stops after TF_TP_PLAN_TIMEOUT with that reason (README). - roce_pick: `select` / `selectFrom` return how the devices were chosen (`resolve` is unchanged for the wire), `joinNames` and `hcaLine` give the NCCL list. Host tests: the port scan over a stub verbs table (state, Ethernet, the IPv4 RoCE v2 GID index), then both ranks through a real bootstrap on the two hosts ashhart#636's receipt reports (modelled, documentation addresses), an explicit rank, and a host without ports.

…idency budget, streamed NVMe tier. zig/src/sessions keeps a chat's KV pages after its request ends: entries are found by a trie over chained page digests, held in RAM within a residency budget (chats and pool bytes), and the least recently used idle chat is parked to an NVMe tier and streamed back on resume. Ranks that split KV run every store call in the same order, so their books stay identical. It complements core/prompt_cache, which snapshots a whole family state; a family that uses it reports the resident bytes as Memory.cache and its budget as Info.prompt_cache_plan. No family uses it yet; zig build test-sessions runs the host tests, the NVMe tier on real files.

ashhart · 2026-10-15T08:56:41Z

Thank you, @jayleaton, for the prompt piece planner, the halt lifecycle and their deterministic host tests.

We are closing this copy because #861 now carries the same planner and lifecycle work as its own review, as we asked on #635. The lifetime and allocation failure fixes are tracked there, and the DeepSeek-V4.1 integration continues under #847.

Please keep going on #861; the family changes can come through that shared core once its gates pass.

Posted by Ash's AI Assistant Synthena

…ost halt Opened.halt came with the closed ashhart#662 and is absent from this stack, so the native build failed to compile at main.zig.

Extract the family-neutral lane changes from #662 onto pinned main, including per-request proposers, row policies, deterministic pricing, prompt-piece lifecycle and the round-boundary halt tests. Ported from a4d95d8657c54adc0ec683eb84df84e22963414f, 55aedb0715266d73f216fc4414de1c5ebc5c4ef0 and 00025f0ab33c319b76fadc1d54b1ed10a1dfe8ea. No sessions, TP or model-family dependency is required.

Extract the family-neutral lane changes from ashhart#662 onto pinned main, including per-request proposers, row policies, deterministic pricing, prompt-piece lifecycle and the round-boundary halt tests. Ported from a4d95d8657c54adc0ec683eb84df84e22963414f, 55aedb0715266d73f216fc4414de1c5ebc5c4ef0 and 00025f0ab33c319b76fadc1d54b1ed10a1dfe8ea. No sessions, TP or model-family dependency is required.

jayleaton and others added 4 commits October 11, 2026 17:13

fix(cuda): correct RDMA VMM capability attribute

faf5901

Use CUDA device attribute 110 for GPUDirect RDMA with VMM support. Attribute 126 reports the memory sync domain count and can incorrectly enable the RDMA allocation flag. Add a regression for the external ABI contract.

jayleaton mentioned this pull request Oct 11, 2026

server: graceful drain and pre-token keepalives (DeepSeek-V4.1 stack 7/14) #663

Closed

3 of 8 tasks

jayleaton added 11 commits October 11, 2026 18:33

TP tests: a drain on rank 0 then its clean stop ends rank 1's loop at…

6b5404d

… the stop word with no fail-fast exit; a rank killed mid-drain still fails the other fast.

TP README: when -Dtp-nvcc, TF_TP_ROCE_HCA, NCCL_IB_HCA, NCCL_SOCKET_I…

062af8b

…FNAME and TF_TP_MASTER are needed, with a two-host example

jayleaton force-pushed the stack/06-lanes branch from a38597d to 19853f9 Compare October 11, 2026 12:27

jayleaton added 5 commits October 11, 2026 21:41

Fix RoCE test smaps lookup for merged mappings

cf19042

jayleaton force-pushed the stack/06-lanes branch from 19853f9 to 006908c Compare October 11, 2026 14:42

ashhart mentioned this pull request Oct 11, 2026

cuda: a keyed graph cache and a device arena, off until a family installs them; failed launches name their kernel (DeepSeek-V4.1 stack 2/14) #635

Closed

3 of 8 tasks

jayleaton added 2 commits October 12, 2026 06:44

jayleaton force-pushed the stack/06-lanes branch from 006908c to e9f9210 Compare October 12, 2026 09:15

This was referenced Oct 14, 2026

Landing next #286

Open

DeepSeek-V4.1 on the native engine: one family, CUDA and Metal (tracker) #847

Open

jayleaton mentioned this pull request Oct 15, 2026

core: extract family-neutral lane planners and halt lifecycle #861

Closed

5 of 8 tasks

ashhart closed this Oct 15, 2026

Navigation Menu

Repository navigation

lanes: a family's round planner drives the lane host with prompt pieces, and a halt ends the rounds at a drain deadline (DeepSeek-V4.1 stack 6/14) - #662

lanes: a family's round planner drives the lane host with prompt pieces, and a halt ends the rounds at a drain deadline (DeepSeek-V4.1 stack 6/14)#662
jayleaton wants to merge 22 commits into
ashhart:mainfrom
jayleaton:stack/06-lanes

jayleaton commented Oct 11, 2026 •

edited

Loading

Uh oh!

ashhart commented Oct 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jayleaton commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this changes

Receipt

Checklist

Merge Danger

Uh oh!

ashhart commented Oct 15, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jayleaton commented Oct 11, 2026 •

edited

Loading