perf(io): read expert slices straight into the cache, with no bounce copy - #143
perf(io): read expert slices straight into the cache, with no bounce copy#143Helldez wants to merge 7 commits into
Conversation
Correction: the +20% above was measured where the bug could not showRunning this in the demo app — long session, several turns — it measured slower than the baseline, not faster:
Reversing the order of the runs changed nothing, so it is neither warm-up nor thermal drift. The fault count rises turn over turn (66 → 332), which is the shape of a leak rather than of a fixed cost, and the process's swap grew 274 → 364 MiB across the session while the flag-off control stayed flat at 272. CauseThe placement moves each layer buffer off the page boundary on purpose. That makes the first and last page of every expert slice shared with the neighbouring expert — where the aligned placement had no partial pages at all. Eviction releases only the pages an entry fully covers, and it must: it may never free a page a neighbour is still using. Under the aligned placement that rule was exact, because "fully covered" meant all of them. Under the shift it leaks two pages per eviction — bytes the cache counts as freed that the kernel still holds. The arithmetic matches the measurement: commit covers 145 pages per slice, release covers 143, and a turn evicts ~5200 entries → ~41 MiB a turn, against a measured swap growth of 9 MiB in the first turn and 44 in the second. FixEviction now also takes a boundary page once the neighbour sharing it is gone — not resident ( Why the original measurement missed itEvery device run behind the +20% was short, single-turn, in a fresh process (24-40 tokens). A leak has no time to accumulate there. Only the app's long session shows it — the trap this project had already recorded once, that a single-shot bench cannot see a reclaim problem. Neither number in this PR is settled. The +20% and the regression were both measured against a leaking build; the headline needs re-taking on a long app session before it means anything. The flag stays off by default. |
…copy O_DIRECT wants the file offset, the length and the buffer address aligned. Expert slices are a whole number of pages long, but a gguf's tensor data does not start on a page boundary: general.alignment is 32 by default, so on the measured model family every expert offset sits a constant 1152 bytes past one. The reader therefore pulled the enclosing aligned window into a per-lane bounce buffer and memcpy'd the payload back by that remainder — over 200 MiB a token of pure shift correction, on a device whose decode is already memory-bound. That shift is only needed because the destination did not share the file's remainder. It can: this engine reserves the per-layer buffers itself and rebinds tensor->data onto them. With --odirect-zero-copy each buffer is placed at an address carrying its own tensor's remainder; every expert inherits it because the per-expert stride is a multiple of the page size, so the aligned window maps onto the buffer in place and the read needs no copy anywhere. The window overhangs its neighbours' slices. That is safe by construction, not by luck: under this placement buffer and file differ by a constant offset, so the overhung bytes receive their own correct file contents, and they land in pages the entry's commit already covers exactly — eviction never releases a page shared with a neighbour, which was already true before this change. Measured on device (Qwen3.6-35B-A3B Q4_0, cache 2000 MiB, overlap, 4 lanes, interleaved cells so thermal drift hits both): +20% tok/s, 3.65 -> 4.40 median, compute residual 0.180 -> 0.143 s/token, identical bytes read (230.77 MiB/token in every cell) and byte-identical generated text with the flag on and off. The phone was warm and its baseline drifted 3.90 -> 3.40 across the run, so the ratio is the claim, not the absolutes; a cool-device confirmation is owed, which is why the flag ships off. This is not specific to any decode feature — every expert read goes through this path, and so does the dense loader's. The host gates cannot prove the path: a tiny test model's per-expert stride is not a multiple of the page size, so the placement declines and G8/G4e show only that the flag is harmless. Rather than let that read as a pass, the engine reports which happened (odirect-zero-copy ON|INERT — n/m layer buffers placed). The device run above is the real proof. Gates 7/7.
…phone The engine flag is useless in the app without a way to turn it on, and the device A/B this change still owes has to be run through the app as well as the CLI. The switch sits under Direct I/O and greys out without it, since O_DIRECT is what forces the bounce copy in the first place. Worded as a pure speed knob, because it is one: unlike expert dropping or route-ahead it is lossless by construction — the same bytes land in the same places, and the generated text is byte-identical with it on and off. It joins the session signature, so flipping it reopens the session rather than silently carrying the old setting into the next generation.
Deciding the zero-copy path from the destination's alignment alone was wrong, and it broke the dense loader: it reads through its own FileReader into separately sized per-tensor buffers, and the aligned window is longer than the tensor it was asked for. The most ordinary case of all triggers it — a page-aligned tensor offset read into a page-aligned buffer matches the remainder test with shift 0, then overruns the buffer by up to a page. On device with --dense-weights ahwb (the app's default, which the earlier CLI runs did not use) this surfaced as `pread failed` and a failed session open. read() now takes allow_in_place, which is a promise: the pages either side of the request, out to the enclosing alignment boundaries, are the caller's to clobber. Only the expert cache can make it, and only for the buffers actually placed for it — recorded per buffer rather than re-derived, because a remainder of zero is a valid placement and must not be confused with a declined one. That distinction was a second latent bug: with the remainder zero the extra page was not reserved, so the last expert's window would have run past the end of the reservation. G9 gates the combination that failed (zero-copy + the anon dense loader). It bites on any model, unlike G8/G4e, whose expert-side placement goes inert on a tiny test model. Verified on device with the exact failing configuration: 120/120 buffers placed, generation byte-identical with the flag on and off, 3.39 -> 4.00 tok/s in the same pair. Gates 7/7.
The zero-copy placement moves each layer buffer off the page boundary on purpose, which makes the first and last page of EVERY expert slice shared with the neighbouring expert. Eviction releases only the pages an entry fully covers — correct, and previously exact, because the page-aligned placement had no partial pages at all. With the shift it leaks two pages per eviction: the cache accounts the bytes as freed while the kernel still holds them. Measured in the app, where a session evicts thousands of entries and lives long enough for it to accumulate: the process's anonymous footprint grew ~40 MiB a turn and went to zram, and the decode paid it back as major faults — 1-7 per token without the flag, 66 on the first turn with it and 332 on the second, rising turn over turn, which is the shape of a leak rather than a cost. Net effect in the app: 5.02-5.58 tok/s without, 4.41-4.57 with. Reversing the run order changed nothing, so it was not warm-up or thermal drift. A boundary page is now released as well, once the neighbour sharing it is gone — `cvalid_` covers demand reads (an entry is valid from the moment it is staged) and `spec_remaining_` covers speculative ones. Both are read without the lanes' mutex, safe in the one direction that matters: staging happens on this thread so nothing raises them concurrently, and a lane can only lower spec_remaining_ toward zero, so a stale read is stale-high and merely defers a release to the next eviction. The first and last expert of a buffer border its own padding, which nobody else owns. Why the earlier device runs missed it: they were short, single-turn, in a fresh process — 24 to 40 tokens, where the leak has no time to accumulate. The app's long session is the only place it shows, which is the trap this project already recorded once: a single-shot bench cannot see a reclaim problem. Gates 7/7 on the host. The claim this branch carries still needs re-measuring in the app, on a long session, before its headline number means anything.
The +20% in this branch was measured with short single-turn CLI runs, which is exactly the regime where a page leak cannot accumulate. In a long app session the same build measured slower than the baseline until the boundary pages were released. Both the win and the regression are now recorded, along with why the earlier measurement could not have caught it, and neither number is presented as settled until a long session re-measures it.
G8 and G9 were already taken, by cache-aware expert dropping and by the prediction probe, so this branch's two gates answered to labels that name a different feature in the same binary's output. A failing "G9" could not be read without opening the file. They become G11 and G12. The index at the top of the file had also stopped describing the file below it: it predates both the drop gates and these, so the map named neither. It now lists them, and says that gate numbers are allocated once and never reused, which is the rule this collision broke.
12d1511 to
4a42df3
Compare
The combination produces wrong bytes with no error reported. Measured against a real model: same revision, same gguf, same flags, correct text on Android and garbage on Windows. Everything on this side is proven correct (requests, destination arithmetic, commit ranges, alignment, the partial-read retry), so the fault is below us; the difference is that in place the DMA target is freshly committed shared memory a compute thread is concurrently reading, where the bounce had a private buffer and a CPU copy ordered against the ready flag. Serial is correct on Windows, so only the overlapped path falls back, and it says so rather than degrading in silence. Tracked in issue #149.
|
Parked, not abandoned. The Windows corruption is tracked in #149 and this branch now carries a guard that falls back to the bounce buffer under --overlap\ on Windows instead of producing wrong bytes silently, so nothing here is unsafe to build. It stays out of v0.19.0: the flag is experimental, the win it claims is owed a re-measurement after the boundary-page fix, and the gate that should protect it (G4e) is still inert on the tiny fixture. Landing it would mean shipping a feature whose only automated coverage proves nothing. |
1 participant
What
--odirect-zero-copy(off by default): expert slices are read straight into the cache buffers, with no bounce buffer and no copy. Measured on device: +20% tok/s, identical bytes read, byte-identical output.This is not a decode feature — every expert read in the engine goes through this path, so it applies to the plain streaming baseline, expert dropping, the prefetchers, everything.
Why the copy existed
O_DIRECT requires the file offset, the transfer length and the buffer address to be aligned. Expert slices are a whole number of pages long, but a gguf's tensor data does not begin on a page boundary —
general.alignmentis 32 by default, and on the model family measured here every expert offset lands a constant 1152 bytes past one:So the reader could not ask the kernel for the payload directly. It pulled the enclosing aligned window — which starts 1152 bytes early — into a per-lane bounce buffer, then
memcpy'd the payload back by that remainder so it landed wheremul_mat_idlooks for it.The copy was therefore never a safety measure. It was a shift correction, costing 200+ MiB a token (~450 MB/token of DRAM traffic once the write side is counted) on a device whose decode is already memory-bound.
Why it can go
The shift is only needed because the destination does not share the file's remainder — and it can. This engine reserves the per-layer buffers itself (
vm_reserve, thentensor->datais rebound), so it controls their addresses. With the flag on, each buffer is placed at an address carrying its own tensor's remainder. Every expert inherits it, because the per-expert stride is a multiple of the page size. The aligned window then maps onto the buffer at the same relative position and the read goes straight in.The overhang, and why it is safe by construction
A read now writes up to
shiftbytes before its slice and the rest of a page after it — bytes belonging to the neighbouring experts. Three facts make that sound rather than lucky:round_down(dst)…round_up(dst+slice)), so no uncommitted page is ever touched. This falls out of the arithmetic, not from a new invariant.A compute thread reading a neighbour's boundary concurrently sees identical values.
Results (device, Qwen3.6-35B-A3B Q4_0, cache 2000 MiB, overlap, 4 lanes)
Interleaved cells, so thermal drift hits both configurations equally:
+20% median (3.65 → 4.40), with the win landing exactly where the mechanism predicts: the compute residual falls 0.180 → 0.143 s/token. Bytes read are identical to the byte in every cell, and the generated text is byte-identical with the flag on and off (greedy, same prompt, same md5) — which is the correctness proof.
The phone was warm and its own baseline drifted 3.90 → 3.40 across the run, so the ratio is the claim, not the absolutes. Note the last and hottest cell (zero-copy, pass b) still beats the first and coolest baseline cell by 12%.
On the host gates
The gates cannot prove this path, and the PR does not pretend otherwise: on a tiny test model the per-expert stride is not a multiple of the page size, so the placement declines and G8 / G4e demonstrate only that the flag is harmless. Rather than let a silent no-op read as a pass, the engine states which happened:
The device run above is the proof. Gates 7/7.
Fallbacks
Back to the bounce whenever the remainder would leave the tensor under-aligned for the compute kernels (below 64 bytes), the per-expert stride is not a multiple of the alignment, or O_DIRECT was refused at open. Buffered mode already read straight into the destination and is untouched.
Scope
Off by default pending a confirmation run on a cool device. Docs:
docs/moe-streaming.md, CHANGELOG under Unreleased. New CSV preamble keyodirect_zero_copy=.