[BlivionIaG]

K/V pool: optional host-resident tier

Draft PR; validation is in progress. Pushed head: 1faf9815e37fd993b5e141eae504d42a555796d4. Standalone core source, based on main 57186386d.

Adds a generic optional host-resident tier to the core K/V pool: canonical host pages, device-frame residency, generation checks and pin-aware eviction. It is disabled by default and is intended to land with its first consumer, #1522.

This PR is independently based on the listed main commit and changes only three core files. Its archived standalone source passes all 48 K/V-pool tests. Family code, CUDA kernels and EXL3 serving integration remain outside this PR.

Rebased onto current main

Rebased onto main 57186386d.

This standalone core change retains upstream pool behavior while adding the freed-page hook.

Validation and remaining gates

On the rebased chain tip #1446 (7c4664dac59d75c7d85e2f5a22d3b5850546e97d) plus the expert branch, host gates passed: zig fmt --check, zig build test (9,566 passed, 55 skipped), test-flashnext-exl3-host (258), test-kernel-contract (150), test-golden, check-native-cuda-host, and the K/V steps (101). These are tip results, not per-head CI results.

The v8 device matrix was completed on the previous rebase onto main 2761c24d, with the same chain code, and is being repeated on the current tip. TP1 and TP2 covered c=1 docs, c=1 code and c=4, with 16,384 input and 1,024 output tokens, drafts 6, graphs and async enabled, actual graph replay witnessed on every rank, no prefix cache and no hidden warmups. Full outputs matched TP1 vs TP2, drafted vs plain, and concurrent vs solo; these device results are not claimed for the current heads.

The matched performance comparison to #1113 and uninstrumented ABAB remain outstanding. Earlier validation and resource results were recorded on pre-rebase sources and do not establish results for the rebased head.

Performance work is deferred on purpose

This chain lands the architecture first: a working, bit-exact base on upstream main (TP=2 equals TP=1, drafted equals plain, prompt rows equal decode rows, NVFP4 bytes unchanged). The performance optimizations of the origin draft (#1113) are not all ported here. We ported only the ones that are exact and cheap; the rest follow in separate PRs once this base is merged, each with its own byte-equality receipt, so nothing in this chain has to be rewritten.

Known gaps against #1113 on the same hardware (numbers in the tables below): deep prefill is slower than #1113's non-exact version, drafted decode and the expert-line count at 2 streams are below it, and the two-stream prose case with drafts on is slower than plain. Candidates for the follow-ups: tile tuning, overlapping the partial all-reduce with attention, trimming the deep-prefill chunk buffers, a cheaper draft probe.

This PR remains a draft while qualification and maintainer acceptance are outstanding. Fixes #1511.

[ashhart]

Thanks @BlivionIaG. This is the tier we hoped for: generic, beside the pool rather than a second pager, with the pool still the only owner of page ids. The random loop, the lockstep digest and the mutants make a good test suite.

It doesn't need slices 1 to 4. The last commit cherry-picks cleanly onto main by itself, and there zig fmt, the lean check (no new findings) and test-kv-pool pass. As we suggested on #1444, please give it its own issue, and rebase this PR onto main with only that commit.

Before it leaves draft:

1. A failed fill leaves its misses mapped. fill unpins on error (zig/src/core/kv_tier.zig:156), but each miss's mapping (:141-142) stays. The family got an error, so it copied nothing. The next pin of that page is then a hit over rows that were never copied in, and kv_tier_test.zig:110 asserts exactly that. Please unmap the frames a failed call mapped, moving their generation on. Add a test that fails today: a failed fill, then the same pages pinned through the rig, with each frame checked against host_page.
2. markDirty (:212) needs only a frame, so a dirty frame can be unpinned. victim skips it (:332), and a lease inside its promise can then get FrameMemory, which the frame promise is meant to rule out. Please require a pin by the lease. Have finish apply the dirty rule to every page it pinned, not only to its own table (:257).

No family uses the tier yet, so we'd land it together with its first user, off by default: slice 6, as #1419 lists it. Device cells can then check tokens against the device-only pool under eviction, and speed.

[ashhart]

Thanks @BlivionIaG. The latest push looks like it covers both points: a failed fill now
unmaps its misses and moves their generation on, with a test that re-pins through the
rig and checks the host rows, and markDirty now needs a pin by the lease, with finish
checking every page it pinned. The branch also looks like it is down to the single
commit on main. Thank you.

One small thing: failedFillReads takes a fault but only runs with none. A mutant
that skips the undo and goes red would match the rest of the suite.

As before, we'd land this together with its first user, slice 6, off by default. Could
you link the issue for the tier here once it's open?