Repository navigation
Conversation
…s where the last one's history ended
…arts and requests that name none The server sends message starts once ashhart#1335 lands, so one conversation's turns can name chunk starts while another's (or an older server's) name none. Each case writes a state, restarts and reads it back from disk with the other kind of request: a state written with starts on the grid serves a request that names none; one at 1000, or at 1024 with its chunks below cut at 300, serves none; a grid state serves named starts that hold the grid's below it and its own position, and no others. Every resume is a fresh pass's bytes.
…arts and requests that name none The server sends message starts once ashhart#1335 lands, so one conversation's turns can name chunk starts while another's (or an older server's) name none. Each case writes a state, restarts and reads it back from disk with the other kind of request: a state written with starts on the grid serves a request that names none; one at 1000, or at 1024 with its chunks below cut at 300, serves none; a grid state serves named starts that hold the grid's below it and its own position, and no others. Every resume is a fresh pass's bytes.
|
Thanks for tracing #1334 so clearly; the later-turn numbers are a big improvement. Resumed equals fresh, and The difficulty is that this is on by default and changes the arithmetic of multi-message prompts. Fresh replies A small one: the commit uses |
…arts and requests that name none The server sends message starts once ashhart#1335 lands, so one conversation's turns can name chunk starts while another's (or an older server's) name none. Each case writes a state, restarts and reads it back from disk with the other kind of request: a state written with starts on the grid serves a request that names none; one at 1000, or at 1024 with its chunks below cut at 300, serves none; a grid state serves named starts that hold the grid's below it and its own position, and no others. Every resume is a fresh pass's bytes.
…arts and requests that name none The server sends message starts once ashhart#1335 lands, so one conversation's turns can name chunk starts while another's (or an older server's) name none. Each case writes a state, restarts and reads it back from disk with the other kind of request: a state written with starts on the grid serves a request that names none; one at 1000, or at 1024 with its chunks below cut at 300, serves none; a grid state serves named starts that hold the grid's below it and its own position, and no others. Every resume is a fresh pass's bytes.
…arts and requests that name none The server sends message starts once #1335 lands, so one conversation's turns can name chunk starts while another's (or an older server's) name none. Each case writes a state, restarts and reads it back from disk with the other kind of request: a state written with starts on the grid serves a request that names none; one at 1000, or at 1024 with its chunks below cut at 300, serves none; a grid state serves named starts that hold the grid's below it and its own position, and no others. Every resume is a fresh pass's bytes.
|
Hi @gilby, thanks again for this, and for the clear trace in #1334. Resuming a chat at the previous turn's history is something we want on CUDA. Following up on the earlier note, two things are needed before we can land it in 1.2.0:
The branch also no longer merges cleanly with main. |
|
Hi @gilby, a friendly check on this one. The main thing from our 27 Oct note is still open: put |
|
Hi @gilby, checking in on this one. It's still the fix #1334 needs on CUDA. The one open point is putting |
|
Thanks @ashhart. Yes, please draft the opt-in on top of my commits and keep me as the author — that's the fastest path to landing, and it matches what you described: |
|
Thanks @gilby, we'll do that. We'll put the opt-in on top of your commit, off by default, rebased onto current main, with you kept as the author, and check it with |
|
@gilby, the opt-in is drafted as you asked, as #2176: your commit with your authorship, rebased onto main, then |
|
/atc exchange ffa9ed5b69d38e0c68c164e676f9db273cfb2b1a2fd1e6115ea7c8779e82d6e8 |
Fixes #1334 (draft until the arithmetic question there is settled).
A CUDA family that also cuts its cold prompt passes at request chunk starts declares
request_starts; for such a family the native host advertises its grid asprefill_step, so the chat server sends message starts and the planned prompt cache keeps and resumes at them. Qwen 27B and its MoE sibling opt in:cuda_prefill.zigalready cuts atstream.chunks(nextCut) and resumes only where a cold pass cuts (coldCut). GLM's CUDA prefill does not read the request's chunks, so it is unchanged.Changes
native/cuda_snaps.zig:requestStep(F)(the family'ssnap_gridwhen it declaresrequest_starts, else 0), with a test.native/cuda.zig:.prefill_step = snaps.requestStep(F)in the served info.qwen3_5/cuda_native.zig,cuda_native_moe.zig:request_starts = true.Receipts (one DGX Spark, GB10 128 GB, driver 580, CUDA 13.0;
TensorFold/Qwen3.8-27B-MLX-4bit@ 22d8d538 +z-lab/Qwen3.8-27B-DFlash2,--parallel 4 --context 262144;mainand this branch alternated in one session)Multi-turn chat, first-token time per turn (s), six turns, ~200-token greedy replies, thinking off:
mainmainEvery reply with the cache on equals the reply with
--prompt-cache-gib 0, on both builds.bench_concurrent --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial, two runs each: 0 unequal, 0 failed. Code 68.8 / 69.0 tok/s at 1 stream and 186.3 / 186.3 at 4; chat 50.0 / 49.9 and 141.4 / 140.7.bench_cache --require-cold --require-reusedoes not qualify on either build (a raw prompt has no message starts; see Qwen 27B on native CUDA: a chat's next turn never resumes at the previous turn, only on the 4,096-row grid #1334).zig build test: 500/500 steps;zig fmtclean;lean_check.pyadds no problems overmain.Arithmetic
A cold pass now also cuts at message starts, so multi-message prompts' outputs differ bitwise from
main's (0 of 12 fresh replies matched), each exact under the contract; a first turn costs ~6% more. Qwen27 on Metal already cuts at request starts.