diff --git a/README.md b/README.md index bbc548d..c3e4ee0 100644 --- a/README.md +++ b/README.md @@ -2,45 +2,225 @@ kern

-**Why does an inference engine need to understand every model it runs?** +

Built by agents. Built for agents.

+ +

+ kern lets agents change the whole inference stack, fast and safely,
+ so every model, every workload and every GPU gets a high-performance engine of its own. +

+ +

+ Website · + Quick start · + Docs +

+ +--- + +## What agents have done with it + +- [**One night, 17% more requests than vLLM.**](docs/vllm-rsi-example.md) Agents optimized Qwen3.8-27B on live Claude Code traffic, three rounds, no human in between. +- [**The same engine in SGLang, 10.5% more requests.**](docs/sglang.md) The overnight manifest, rewritten for SGLang's memory layout. +- [**A new model family in one session.**](docs/qwen38-bringup.md) Qwen3.8-27B plus its speculative draft in under 2.5 hours, one human intervention. +- [**A kernel set from six agents in parallel.**](docs/k3-kernel-abi.md) K3 decode kernels against a written ABI, step time 37.2 → 29.3 ms. +- [**An engine per GPU.**](docs/registry.md) DeepSeek-V4.1-Flash ships separate manifests for GB300 and HGX B300. +- [**Full-size models across GPUs and trays.**](docs/serve.md) DeepSeek-V4.1-Flash at EP4, full-size K3 prefill across eight pipeline stages. + +## Own your stack. Today's engines are million-line programs that carry, in one codebase: ``` -models × precisions × GPUs × parallelism × decoding tricks × … +models × precisions × GPUs × parallelism × decoding tricks × workloads ``` -Every new factor multiplies the matrix. The engine absorbs all of it. -To run *one* model, you first teach an engine about *all* of them — -and one kernel change re-certifies the entire product. +The two biggest open engines: 1.4M and 2.0M lines, 280 and 247 model files. -It was never supposed to work this way. - ---- +Say you only want K3 to be fast on an NVL72. In a shared engine that is a +change to code that also serves every other model on every other GPU, and +there are three ways to make it: -## A model is not code to merge. It is a program to verify. +- **Upstream.** Your PR joins 10,000 others open in those two. Four in five + have no human review. +- **Fork.** Skip rebases and miss every new model and kernel. Take them and + fight conflicts; even a clean one ships thousands of PRs never run on your + GPUs. +- **Hand it to an agent.** Which kernel runs? If the arch is this, and the + head count is that, and the dtype is something else, unless parallelism is + on, or a flag says otherwise. Good luck. -A model ships as three files: +kern splits engine from model. The engine contains no model and rarely +changes. The model, parallelism included, ships as three things: ``` -manifest.json one typed declaration — buffers, ops, programs +manifest.json one typed declaration: buffers, states, ops, programs, parallelism kernels/ compiled device code, from anywhere -weights the model's own checkpoint; the manifest says which tensor lands where +weights the model's own Hugging Face checkpoint ``` -**A manifest is one point in that exponential space — declared, verified, -shipped alone.** The combination lives in the artifact, not the engine. +**A model is not code to merge. It is a package you own.** -The runtime reads the manifest the way a compiler reads source: -verify everything, refuse anything inconsistent, then execute blindly. -It contains no model. It never will. +Your K3-on-NVL72 engine is your repo, like +[kern-k3-pp-prefill](https://github.com/pegainfer-project/kern-k3-pp-prefill). +No review queue, no rebase, no "it depends." For an agent, that repo is the +whole world. -**One manifest. Any kernel. Zero trust.** +## Architecture -## Proof +![kern architecture: your package is verified, then run inside a host or examined by the harness](docs/figures/architecture.svg) + +### The manifest + +Everything an engine would need to know about a model is written down in the +manifest, so the runtime does not have to know it. A program is a list of +calls. Each call binds an op to buffers and vars, and each op is a typed +signature plus the launches that implement it: + +```json +"ops": { + "embedding": { + "params": ["in buffer", "in buffer", "out buffer", "i32"], + "impl": { "launches": [{ "module": "embedding", "entry": "kern_embedding_i64_bf16", + "grid": ["rows", 1, 1], "block": [256, 1, 1], … }] } } }, +"programs": { + "decode": { "graph": true, "calls": [ + { "op": "embedding", "args": [{"buf": "token_ids"}, {"buf": "embed"}, {"buf": "hidden"}, {"var": "rows"}] }, + … ] } } +``` + +A module is any compiled device code: vendored from FlashInfer or TRT-LLM, +written by hand, or written by an agent. It is pinned by SHA-256 and fetched +from a path, a URL or the Hugging Face hub. Swapping a kernel means editing +one launch. + +**Memory** comes in two kinds, and K3 shows both: + +```json +"buffers": { + "token_ids": { "dtype": "i64", "shape": ["rows"], "kind": "input" }, + "embed": { "dtype": "bf16", "shape": [163840, 7168], "kind": "weight", + "bind": [{ "tensor": "language_model.model.embed_tokens.weight" }] }, + "stats": { "dtype": "i32", "shape": [224], "kind": "workspace" } +}, +"states": { + "kv": { "bytes_per_token": 1152 }, + "kda": { "bytes_per_seq": 19537920 } +} +``` -One line of the manifest points at a kernel package on the Hugging Face -hub — the stock torch extension the PyTorch ecosystem uses: +A buffer is a tensor whose kind says where it comes from: the caller +supplies an `input`, a `weight` binds by name to a tensor in the +Hugging Face checkpoint, and `workspace` is scratch that lasts one call. +A state is memory that outlives a step, declared only by its size. K3's MLA +cache takes 1,152 bytes per token and its KDA linear-attention state takes +19.5 MB per sequence. The runtime pages the first and hands out slots of the +second, both from one pool, and never learns that one is attention and the +other a recurrence. + +**Shapes** come from vars. `rows` in the shapes and launch grids above is +one, along with sequences and context length, and each has a bound. Every +buffer size and launch grid is checked at those bounds, and a call outside +them is rejected before it launches. + +**Parallelism and serving** are declared too. Topology names rank groups, +and a `peer` buffer holds the other ranks' addresses, so an all-reduce or an +all-to-all is a kernel in the manifest. A new parallel layout is a new +manifest, not a runtime change. The serving protocol says which buffers carry +tokens, slots and sequence lengths, and which programs run per batch. Hosts +read those roles and never a model's name. + +Because all of this is data, `verify` checks the whole manifest at once in +milliseconds and reports every problem, not just the first. That is the loop +an agent iterates against. + +### The runtime + +`kern-runtime` loads a verified manifest, allocates, binds weights, lowers +each program to a flat launch list and replays it, captured as a CUDA graph +per shape bucket where it can. The binary links no CUDA library; it dlopens +the driver and cuBLAS on first use. Nothing in it branches on a model. + +`kern-pool` makes every decision about state memory as pure host code and +returns it as a plan the runtime executes, so the bookkeeping is tested +without a GPU. It does two things. + +![kern-pool page sharing: a lease writes its pages, retiring freezes them in place as a checkpoint, and a request with the same prefix shares the whole pages and copies only the partial one](docs/figures/pool.svg) + +**It shares prefixes by page.** A request leases the pages its worst case +needs and a state slot, all or nothing. When it ends, its pages freeze in +place as a checkpoint. A later request with the same prefix shares the +whole pages and gets a copy of only the page the prefix ends inside, so +every page has at most one writer. A token radix tree finds the longest +checkpoint for a prompt; under pressure, checkpoints park in host memory and +wake straight into a new lease. + +![kern-pool memory budget: a static split leaves a sixth request waiting while KV memory sits free; kern remaps free pages' physical chunks under the slot range and runs it](docs/figures/pool_budget.svg) + +**It keeps one budget for all state.** Every state reserves a virtual +address range once, and one pool of physical chunks backs them all through +CUDA VMM. Leasing and freeing touch only free lists, and a freed page stays +a page. Only when slots run short while pages sit free, or the reverse, does +the pool remap: the free pages' chunks are unmapped and mapped under the +slot range. Kernels still see one contiguous array per state, so nothing is +copied, no kernel changes, memory never fragments, and the same chunks can +be exported to ranks on other trays. The manifest's `seqs` bound sizes one +step's batch; how many sequences hold state is up to the budget, not a +number fixed at startup. + +### The harness + +`kern test` takes a reference manifest A and a candidate B. It diffs them +call by call, then records A on a seeded workload, keeping each span's +inputs and the bytes it wrote. B replays span by span from A's own inputs, +so each comparison covers exactly one span's write set, against a noise +floor measured by replaying A against itself. End to end it compares logits +by per-row KL and tells near-ties from real flips by the top-1/top-2 margin. +The verdict is one line and an exit code: 0 pass, 1 fail, 2 inconclusive. + +`kern bench` takes a workload of shapes (sequences × rows × context) and +attributes time to every call inside the real program; `--isolate` measures +a single call on its own, cold and warm. A server's traced steps become a +workload directly, so a change is priced in GPU seconds on the traffic +actually served. `kern cut` slices a manifest into pipeline stages by +liveness, each stage a manifest whose inputs are the previous stage's +outputs. + +Everything is deterministic: no clocks, no randomness, no dependence on +iteration order. The same manifest and inputs give the same bytes, so +running twice and diffing is a test. + +### Hosts + +`kern-serve` is kern's own server: continuous batching, an OpenAI-compatible +API, a prefix cache, sessions that sleep to host memory, and EP, TP and +pipeline stages across trays. kern also runs as the model inside vLLM or +SGLang: the host allocates the KV and state pools, kern checks their layout +against the manifest's `host` states, and launches enqueue on the host's +stream, inside its CUDA graphs. Scheduling, caching and sampling stay with +the host; the forward pass is the manifest. + +Details: [manifest](docs/manifest.md) · [runtime](docs/runtime.md) · +[pool](docs/pool.md) · [test](docs/test.md) · [bench](docs/bench.md) · +[serve](docs/serve.md) · [vLLM](docs/vllm.md) · [SGLang](docs/sglang.md) + +## Agents change the whole stack + +Everything that makes an engine fast lives in the manifest, so it is all +fair game: which kernel runs each op, where the fusion boundaries are, how a +step maps to launches, how the model splits across GPUs, how speculative +decoding composes its rounds. + +**Fast.** A changed manifest is type-checked in milliseconds. `kern test` +replays it against a reference span by span and returns a verdict in +seconds. `kern bench` says where the time goes on the shapes your service +actually sees, not on one representative batch. + +**Safe.** Zero trust: every kernel is pinned by SHA-256, every buffer it +touches is declared, and nothing ships without a verdict against the +reference. The engine stays the same, so a change can only break the +manifest it lives in. + +One line of a manifest can point at a kernel package on the Hugging Face hub: ```diff "silu_mul": { "params": ["out buffer", "in buffer"], @@ -54,29 +234,7 @@ hub — the stock torch extension the PyTorch ecosystem uses: ``` A runtime with no torch and no Python fetched it, verified it, ran it. -Output: byte-identical. Calls touched: zero. - -And: - -- The entire runtime is **under 3,000 lines of Rust**. -- Speculative decoding took **six programs and zero new kernels** — composed, not implemented. -- **92%** of vLLM's decode throughput, **37×** faster prefill than the naive path. *(Qwen3-4B · GB300 · bs=1)* -- A second model family — **Qwen3.8-27B** (hybrid linear attention, 64 layers) plus its DFlash2 speculative draft — cost the runtime and schema **49 lines**. Everything model-specific landed in a 1.4k-line generator and six kernels under 150 lines; decode 81 vs 95 tok/s, speculative 178 vs 176. *(timeline: [docs/qwen38-bringup.md](docs/qwen38-bringup.md))* - -## The loop - -Machines write kernels now. Shipping one still takes a human review cycle. - -Here, a kernel change is not an engine change: - -``` -swap the impl → verify (ms) → byte-diff (s) → shipped -``` - -No PR. No review queue. No CI across every model. -The loop runs unattended. The engine goes back to being an engine. - ---- +Output: byte-identical. ## Try it @@ -85,65 +243,38 @@ curl -fsSL https://kern-baa.pages.dev/install.sh | sh # Linux x86_64 / aarch64 kern --version # kern 0.2.3 (, cuda 13.0) ``` -Then the [quick start](https://kern-baa.pages.dev/docs/getting-started/): +The [quick start](https://kern-baa.pages.dev/docs/getting-started/) runs Qwen3.8-27B from the published kernels and the checkpoint you already have, -plain and with its DFlash2 draft, in four commands. +plain and with its DFlash2 draft, in four commands. The binary links no CUDA +library; it needs an NVIDIA driver for CUDA 13 (r580+) and cuBLAS 13 on the +loader path. `KERN_VERSION` pins a release, `KERN_INSTALL_DIR` picks the +directory (default `~/.local/bin`). -The binary links no CUDA library; it dlopens the driver and cuBLAS at -first use, so it needs an NVIDIA driver for CUDA 13 (r580+) and cuBLAS 13 -on the loader path, nothing else. `KERN_VERSION=v0.2.3` pins a release, -`KERN_INSTALL_DIR` picks the directory (default `~/.local/bin`). How a -release is cut and gated: [docs/release.md](docs/release.md). - -Or from source: +From source, with the fixture targets in `kern.toml`: ```bash cargo build --release - -# kern.toml at the repo root names the fixture target (manifest, reference, -# kernels dir, weights); every flag can still override it. -./target/release/kern run --steps 320 -./target/release/kern run qwen3-4b-dspark --steps 320 # speculative decoding: the manifest's 7-row round, same runtime - -# the loop: evidence for a kernel swap — diff, tap a seeded workload once -# (random tokens, multi-chunk prefill, N decode steps), then per cut: noise -# floor, bit-diff, fuzz around the tap; end-to-end logits are the verdict; -# eager/TPOT/sweep timing. ~10 s, one line per fact, the last line is the -# verdict (exit 0 PASS / 1 FAIL / 2 INCONCLUSIVE); --json for one object -./target/release/kern test qwen3-4b +./target/release/kern run qwen3-4b --steps 320 # generate +./target/release/kern test qwen3-4b # verdict for a change: exit 0 PASS / 1 FAIL / 2 INCONCLUSIVE +./target/release/kern bench qwen3-4b --workload tools/profiles/quick.toml --out quick.json +./target/release/kern server qwen3-4b --port 8000 # OpenAI-compatible server ``` -Serve a configured target with the independent HTTP server: +Inside vLLM, the manifest is the model and vLLM keeps everything else: ```bash -cargo build --release -p kern-serve # or use the kern-serve the release installs -KERN_SERVE_BIN="$PWD/target/release/kern-serve" \ - ./target/release/kern server qwen3-4b --port 8000 +(cd crates/kern-py && maturin build --release) && pip install target/wheels/kern-*.whl +pip install python/kern_vllm +hf download Pegainfer/kern-qwen38-sm103 --local-dir kern-qwen38 +KERN_MANIFEST=kern-qwen38/manifests/qwen3.8-27b-vllm.json KERN_KERNELS=kern-qwen38/cubins \ +vllm serve Qwen/Qwen3.8-27B --hf-overrides '{"architectures": ["KernForCausalLM"]}' ``` -`kern server` resolves the target's manifest, kernels and weights from -`kern.toml`, then forwards server arguments unchanged. Explicit `--manifest`, -`--kernels` or `--weights` replace the corresponding target defaults. -`KERN_SERVE_BIN` selects the executable; otherwise kern looks beside itself, -then on `PATH`. On Unix the server replaces the CLI process, preserving signals -and exit status. Use `kern server -- --help` for the server's flags. -The frontend picks the chat renderer from the checkpoint's `model_type` -(`deepseek_v41` included; anything else renders through the HF chat -template). The V4.1 renderer matches the released text encoding for system -messages, multi-turn history and thinking; it accepts `low`, `high`, `xhigh` -and `max` reasoning effort, and integer budgets from 1 to 100 as -`chat_template_kwargs.reasoning_effort`. - +SGLang works the same way through `python/kern_sglang`: [docs/sglang.md](docs/sglang.md). `kern --help` lists the flags; `crates/kern-run/src/config.rs` -documents `kern.toml` (targets are names you pick — kern reads no meaning -into them; anything the manifest already knows stays out of it). Logs go -to stderr (`RUST_LOG`); stdout carries the generated text or the report. -The pipeline that produces `kernels/` and `weights/` from a live vLLM -process is in [docs/runtime.md](docs/runtime.md) (`kern kernels` drives -it from `kern.toml`); what `kern test` measures and how it decides is in -[docs/test.md](docs/test.md). +documents `kern.toml`. -## The contract +## Inside The wire format is one JSON Schema, generated from the code and golden-checked in CI: @@ -152,30 +283,18 @@ golden-checked in CI: | Path | What it is | | --- | --- | -| `crates/kern-manifest` | Schema + verifier (pure, no CUDA) | -| `crates/kern-runtime` | The executor: fetch, verify, replay, CUDA graphs | -| `crates/kern-test` | The `kern test` harness: diff, tap, noise, fuzz, perf, verdict over a `Side` (pure, no CUDA) | -| `crates/kern-run` | The `kern` binary: `run` (generation), `test` (the harness over `Runtime`), `kernels` | -| `examples/` | Generated manifests — the artifact a provider ships (`*-silu-mined.json` is the `kern test` fixture) | -| `docs/` | [design](docs/design.md) · [manifest](docs/manifest.md) · [kernel mining](docs/kernel-mining.md) · [runtime](docs/runtime.md) · [test](docs/test.md) · [spec decode](docs/spec-decode.md) · [roadmap](docs/roadmap.md) · [release](docs/release.md) | - -**Website:** [kern-baa.pages.dev](https://kern-baa.pages.dev/) - -### Named constants (manifest v4) - -The optional `constants` table is recommended for readable model dimensions. -Use C-style `UPPER_SNAKE_CASE` names: - -```json -"constants": {"VOCAB_SIZE": 151936, "HIDDEN_SIZE": 2560} -``` - -Use `"shape": ["seqs", "VOCAB_SIZE"]` or `{"i32": "HIDDEN_SIZE"}`. -Names work in numeric fields (including `i64`, `f32`, expressions, capacities -and byte offsets), except `schema_version`, which remains `5`. Values must be -numeric literals; aliases and constant expressions are not supported. Names -must not overlap `vars`. The loader expands references before typed validation, -so destination types and ranges still apply. String-only fields such as `buf` -are unaffected. Serialization emits resolved literals. The public wire schema -is generated by `kern_manifest::json_schema()` from the literal Rust types plus -these reference alternatives. +| `crates/kern-manifest` | Schema and verifier (pure, no CUDA) | +| `crates/kern-pool` | KV pages, state slots, prefix cache, host tier (pure, no CUDA) | +| `crates/kern-runtime` | The executor: fetch, verify, allocate, CUDA graphs | +| `crates/kern-test` | The `kern test` harness: diff, tap, noise floor, fuzz, perf, verdict (pure, no CUDA) | +| `crates/kern-run` | The `kern` binary: `run`, `test`, `bench`, `server`, `verify`, `cut`, `kernels` | +| `crates/kern-serve` | Continuous batching and the OpenAI-compatible endpoint | +| `crates/kern-py`, `python/` | `import kern`, and the vLLM and SGLang plugins | +| `tools/` | Capture, kernel mining, manifest generators: all model knowledge lives here | +| `examples/` | Generated manifests, the artifact a provider ships | + +Design and records: [design](docs/design.md) · [manifest](docs/manifest.md) +· [runtime](docs/runtime.md) · [test](docs/test.md) · [bench](docs/bench.md) +· [serve](docs/serve.md) · [vLLM](docs/vllm.md) · [SGLang](docs/sglang.md) +· [registry](docs/registry.md) · [roadmap](docs/roadmap.md) +· [lessons](docs/lessons.md) · [release](docs/release.md) diff --git a/docs/figures/architecture.py b/docs/figures/architecture.py new file mode 100644 index 0000000..6659b65 --- /dev/null +++ b/docs/figures/architecture.py @@ -0,0 +1,68 @@ +"""The README architecture figure: python3 docs/figures/architecture.py writes architecture.svg beside it.""" +from pathlib import Path +W, H = 960, 464 +INK, BODY, MUTED, LINE, SOFT = "#0f172a", "#475569", "#64748b", "#94a3b8", "#e2e8f0" +ACC, ACC_INK, ACC_BG = "#ea580c", "#9a3412", "#fff7ed" +out = [] +def el(s): out.append(s) +def text(x, y, s, size=13, weight=400, fill=BODY, anchor="start", spacing=None): + ls = f' letter-spacing="{spacing}"' if spacing else "" + el(f'{s}') +def rect(x, y, w, h, r=10, fill="#ffffff", stroke=SOFT, width=1.5, dash=None): + d = f' stroke-dasharray="{dash}"' if dash else "" + el(f'') +def card(x, y, w, h, title, lines, tsize=15): + rect(x, y, w, h) + cx = x + w / 2 + text(cx, y + h / 2 - 8 * len(lines) + 4, title, tsize, 600, INK, "middle") + for i, l in enumerate(lines): + text(cx, y + h / 2 - 8 * len(lines) + 24 + 17 * i, l, 12.5, 400, BODY, "middle") +def arrow(d, end): + ex, ey = end + el(f'') + el(f'') + +el(f'') +el(f'') +el('') + +# the package +rect(32, 56, 232, 400, 14, ACC_BG, ACC, 1.5, "6 5") +text(52, 86, "YOUR PACKAGE", 11.5, 700, ACC_INK, spacing="1.2") +for i, (t, ls) in enumerate([("manifest.json", ["typed program: buffers,", "states, ops, programs"]), + ("kernels/", ["any compiled code,", "pinned by SHA-256"]), + ("checkpoint", ["Hugging Face weights,", "bound tensor by tensor"])]): + card(52, 104 + 116 * i, 192, 96, t, ls) + +# verify +arrow("M264,256 H310", (310, 256)) +rect(316, 204, 164, 104, 12, "#1e293b", "#1e293b") +text(398, 244, "verify", 16, 600, "#ffffff", "middle") +text(398, 266, "kern-manifest", 12.5, 400, "#cbd5e1", "middle") +text(398, 284, "every diagnostic, in ms", 12.5, 400, "#cbd5e1", "middle") + +# fan out +arrow("M480,256 H506 V172 H526", (526, 172)) +arrow("M480,256 H506 V390 H526", (526, 390)) + +# run +rect(532, 56, 396, 248, 14, "#f8fafc", SOFT) +text(552, 86, "RUN", 11.5, 700, MUTED, spacing="1.2") +card(552, 100, 176, 96, "kern-runtime", ["allocate · lower", "replay as CUDA graphs"]) +card(744, 100, 164, 96, "kern-pool", ["pages · slots · leases", "prefix · host tier"]) +el(f'') +text(552, 224, "HOSTED IN", 10.5, 700, MUTED, spacing="1.2") +for i, h in enumerate(["kern-serve", "vLLM", "SGLang"]): + x = 552 + 122 * i + rect(x, 236, 112, 48, 8, "#f1f5f9", SOFT) + text(x + 56, 265, h, 13.5, 600, "#1e293b", "middle") + +# harness +rect(532, 324, 396, 132, 14, "#f8fafc", SOFT) +text(552, 354, "HARNESS", 11.5, 700, MUTED, spacing="1.2") +for i, (t, l) in enumerate([("kern test", "A vs B, verdict"), ("kern bench", "time per call"), ("kern cut", "pipeline stages")]): + card(552 + 122 * i, 366, 112, 72, t, [l], 14) + +el('') +el('') +(Path(__file__).parent / "architecture.svg").write_text("\n".join(out) + "\n") diff --git a/docs/figures/architecture.svg b/docs/figures/architecture.svg new file mode 100644 index 0000000..c315d3a --- /dev/null +++ b/docs/figures/architecture.svg @@ -0,0 +1,58 @@ + + + + +YOUR PACKAGE + +manifest.json +typed program: buffers, +states, ops, programs + +kernels/ +any compiled code, +pinned by SHA-256 + +checkpoint +Hugging Face weights, +bound tensor by tensor + + + +verify +kern-manifest +every diagnostic, in ms + + + + + +RUN + +kern-runtime +allocate · lower +replay as CUDA graphs + +kern-pool +pages · slots · leases +prefix · host tier + +HOSTED IN + +kern-serve + +vLLM + +SGLang + +HARNESS + +kern test +A vs B, verdict + +kern bench +time per call + +kern cut +pipeline stages + + diff --git a/docs/figures/pool.py b/docs/figures/pool.py new file mode 100644 index 0000000..14e931f --- /dev/null +++ b/docs/figures/pool.py @@ -0,0 +1,127 @@ +"""The README pool figure: python3 docs/figures/pool.py writes pool.svg beside it.""" +from pathlib import Path +W, H = 960, 476 +INK, BODY, MUTED, LINE, SOFT = "#0f172a", "#475569", "#64748b", "#94a3b8", "#e2e8f0" +ACC, ACC_INK, ACC_FILL = "#ea580c", "#9a3412", "#fed7aa" +DARK, BAND = "#1e293b", "#f1f5f9" +out = [] +def el(s): out.append(s) +def text(x, y, s, size=13, weight=400, fill=BODY, anchor="start", spacing=None, mono=False): + ls = f' letter-spacing="{spacing}"' if spacing else "" + ff = ' font-family="ui-monospace, SFMono-Regular, Menlo, monospace"' if mono else "" + el(f'{s}') +def rect(x, y, w, h, r=6, fill="#ffffff", stroke=SOFT, width=1.5, dash=None): + d = f' stroke-dasharray="{dash}"' if dash else "" + el(f'') +def line(d, stroke=MUTED, width=2, dash=None): + s = f' stroke-dasharray="{dash}"' if dash else "" + el(f'') +def down(x, y0, y1, stroke=ACC): + line(f"M{x},{y0} V{y1}", stroke, 2.5) + line(f"M{x-6},{y1-9} L{x},{y1} L{x+6},{y1-9}", stroke, 2.5) +def pill(cx, cy, s, fill="#ffffff", ink=MUTED): + w = 7 * len(s) + 16 + rect(cx - w / 2, cy - 10, w, 20, 10, fill, fill, 0) + text(cx, cy + 4, s, 11.5, 600, ink, "middle") + +BW, BH, GAP, X0 = 60, 40, 12, 444 +SX, SW = 820, 108 +def px(i): return X0 + (BW + GAP) * i +def mid(i): return px(i) + BW / 2 +SMID = SX + SW / 2 + +STYLE = {"lease": (ACC_FILL, ACC, ACC_INK), "frozen": (DARK, DARK, "#ffffff")} +def obj(x, y, w, label, style, part=1.0): + if style in ("empty", "freed"): + stroke, ink, weight = (ACC, ACC_INK, 600) if style == "empty" else (LINE, MUTED, 400) + rect(x, y, w, BH, fill="#ffffff", stroke=stroke, dash="4 4") + text(x + w / 2, y + 25, label, 12.5, weight, ink, "middle") + return + fill, stroke, ink = STYLE[style] + if part >= 1: + rect(x, y, w, BH, fill=fill, stroke=stroke) + text(x + w / 2, y + 25, label, 12.5, 600, ink, "middle") + return + rect(x, y, w, BH, fill="#ffffff", stroke=stroke) + el(f'') + rect(x, y, w, BH, fill="none", stroke=stroke) + text(x + w * part / 2, y + 25, label, 12.5, 600, ink, "middle") +def swatch(x, y, style): + if style == "empty": + rect(x, y, 16, 16, 4, "#ffffff", ACC, 1.5, "3 3") + else: + rect(x, y, 16, 16, 4, STYLE[style][0], STYLE[style][1]) + +el(f'') +el(f'') + +text(32, 42, "A and B share a prompt prefix", 19, 700, INK) +text(32, 64, "What kern-pool does with their KV memory, in three steps.", 12.5, 400, BODY) +x = 32 +for style, label in [("lease", "written by its lease"), ("frozen", "frozen: read-only, shareable"), ("empty", "leased, still empty")]: + swatch(x, 84, style) + text(x + 22, 97, label, 11.5, 400, BODY) + x += 22 + 7 * len(label) + 20 + +Y0, RH = 148, 108 +def ry(r): return Y0 + RH * r +ROWS = [ + ("lease", "LEASE A", "A takes pages for its worst case", "plus one slot, all or nothing."), + ("retire", "CHECKPOINT", "A ends inside p3. Nothing moves:", "its pages freeze where they are."), + ("restore", "LEASE B", "B starts at the end of A's prefix", "and keeps writing after it."), +] +for r, (op, who, c1, c2) in enumerate(ROWS): + y = ry(r) + text(32, y + 36, str(r + 1), 38, 700, LINE) + text(76, y + 13, op, 15, 700, INK, mono=True) + text(76, y + 33, c1, 12.5) + text(76, y + 50, c2, 12.5) + text(X0 - 16, y + 25, who, 11.5, 700, MUTED, "end", "1.2") + +y = ry(0) - 12 +line(f"M{px(0)},{y + 4} V{y} H{px(0) + BW} V{y + 4}", LINE, 1.5) +text(px(0), y - 8, "a page = 784 tokens of KV", 11.5, 400, BODY) +line(f"M{SX},{y + 4} V{y} H{SX + SW} V{y + 4}", LINE, 1.5) +text(SX + SW, y - 8, "a slot = one sequence's state", 11.5, 400, BODY, "end") + +g1, g2 = ry(0) + BH, ry(1) + BH +band = (px(0) - 8, ry(1) - 8, px(2) + BW - px(0) + 16, RH + BH + 16) +rect(*band, 12, BAND, "#cbd5e1", 1.5) + +for i in range(4): line(f"M{mid(i)},{g1 + 4} V{ry(1) - 4}", LINE, 1.5, "2 4") +line(f"M{SMID},{g1 + 4} V{ry(1) - 4}", LINE, 1.5, "2 4") +pill((mid(0) + mid(3)) / 2, (g1 + ry(1)) / 2, "same pages, frozen in place") +pill(SMID, (g1 + ry(1)) / 2, "handed over") +pill(mid(4), (g1 + ry(1)) / 2, "to the pool") + +for i in range(3): line(f"M{mid(i)},{g2 + 4} V{ry(2) - 4}", LINE, 1.5, "2 4") +pill((mid(0) + mid(2)) / 2, (g2 + ry(2)) / 2, "shared, zero copy", BAND, INK) +for cx in (mid(3), SMID): + down(cx, g2 + 5, ry(2) - 5) + text(cx + 10, (g2 + ry(2)) / 2 + 4, "copy", 12.5, 700, ACC_INK) + +y = ry(0) +for i in range(3): obj(px(i), y, BW, f"p{i}", "lease") +obj(px(3), y, BW, "p3", "lease", 0.5) +obj(px(4), y, BW, "p4", "empty") +obj(SX, y, SW, "slot s1", "lease") + +y = ry(1) +for i in range(3): obj(px(i), y, BW, f"p{i}", "frozen") +obj(px(3), y, BW, "p3", "frozen", 0.5) +obj(px(4), y, BW, "freed", "freed") +obj(SX, y, SW, "slot s1", "frozen") + +y = ry(2) +for i in range(3): obj(px(i), y, BW, f"p{i}", "frozen") +obj(px(3), y, BW, "p5", "lease", 0.5) +obj(px(4), y, BW, "p6", "empty") +obj(SX, y, SW, "slot s2", "lease") + +y = ry(2) + BH + 32 +line(f"M32,{y} H928", SOFT, 1) +text(32, y + 30, "THE RULE", 11.5, 700, ACC_INK, spacing="1.2") +text(116, y + 30, "Whole pages are shared, the partial page is copied, and only a live lease writes.", 14.5, 600, INK) + +el('') +(Path(__file__).parent / "pool.svg").write_text("\n".join(out) + "\n") diff --git a/docs/figures/pool.svg b/docs/figures/pool.svg new file mode 100644 index 0000000..672bd18 --- /dev/null +++ b/docs/figures/pool.svg @@ -0,0 +1,98 @@ + + +A and B share a prompt prefix +What kern-pool does with their KV memory, in three steps. + +written by its lease + +frozen: read-only, shareable + +leased, still empty +1 +lease +A takes pages for its worst case +plus one slot, all or nothing. +LEASE A +2 +retire +A ends inside p3. Nothing moves: +its pages freeze where they are. +CHECKPOINT +3 +restore +B starts at the end of A's prefix +and keeps writing after it. +LEASE B + +a page = 784 tokens of KV + +a slot = one sequence's state + + + + + + + +same pages, frozen in place + +handed over + +to the pool + + + + +shared, zero copy + + +copy + + +copy + +p0 + +p1 + +p2 + + + +p3 + +p4 + +slot s1 + +p0 + +p1 + +p2 + + + +p3 + +freed + +slot s1 + +p0 + +p1 + +p2 + + + +p5 + +p6 + +slot s2 + +THE RULE +Whole pages are shared, the partial page is copied, and only a live lease writes. + diff --git a/docs/figures/pool_budget.py b/docs/figures/pool_budget.py new file mode 100644 index 0000000..9697ff3 --- /dev/null +++ b/docs/figures/pool_budget.py @@ -0,0 +1,101 @@ +"""The README pool budget figure: python3 docs/figures/pool_budget.py writes pool_budget.svg beside it.""" +from pathlib import Path +W, H = 960, 466 +INK, BODY, MUTED, LINE, SOFT = "#0f172a", "#475569", "#64748b", "#94a3b8", "#e2e8f0" +ACC, ACC_INK, ACC_FILL, ACC_BG = "#ea580c", "#9a3412", "#fed7aa", "#fff7ed" +DARK, FAINT = "#1e293b", "#cbd5e1" +out = [] +def el(s): out.append(s) +def text(x, y, s, size=13, weight=400, fill=BODY, anchor="start", spacing=None): + ls = f' letter-spacing="{spacing}"' if spacing else "" + el(f'{s}') +def rect(x, y, w, h, r=5, fill="#ffffff", stroke=SOFT, width=1.5, dash=None): + d = f' stroke-dasharray="{dash}"' if dash else "" + el(f'') +def line(d, stroke=MUTED, width=2, dash=None): + s = f' stroke-dasharray="{dash}"' if dash else "" + el(f'') + +P, BW, BH = 44, 38, 34 +def cx(x0, i): return x0 + P * i + P / 2 + +def slot(x, y, label, kind): + fill, stroke, ink, dash = {"held": (DARK, DARK, "#ffffff", None), "new": (ACC, ACC, "#ffffff", None), + "void": ("none", FAINT, MUTED, "4 3")}[kind] + rect(x - BW / 2, y, BW, BH, 5, fill, stroke, 1.5, dash) + if label: text(x, y + BH / 2 + 5, label, 13, 700, ink, "middle") + +def pages(x, y, kind): + if kind in ("void", "gone"): + rect(x - BW / 2, y, BW, BH, 5, "none", ACC if kind == "gone" else FAINT, 1.5, "4 3") + return + rect(x - BW / 2, y, BW, BH, 5, "#ffffff", LINE, 1.5) + for k in range(4): + rect(x - BW / 2 + 3 + 8 * k, y + 4, 6, BH - 8, 1.5, LINE if kind == "held" else "#ffffff", LINE, 1) + +def chunk(x, y, moved=False): + rect(x - BW / 2, y, BW, 26, 4, ACC_BG if moved else "#f1f5f9", ACC if moved else LINE, 2 if moved else 1.5) + +el(f'') +el(f'') + +# a static split: same twelve chunks of memory, the line drawn once +X0, Y0 = 40, 50 +text(X0, 34, "A STATIC SPLIT · FIXED AT STARTUP", 11.5, 700, MUTED, spacing="1.2") +for i in range(5): slot(cx(X0, i), Y0, str(i + 1), "held") +for i, k in enumerate(["free"] * 4 + ["held"] * 3): pages(cx(X0, 5 + i), Y0, k) +fx = X0 + 5 * P +line(f"M{fx},{Y0 - 8} V{Y0 + BH + 8}", INK, 3) +text(cx(X0, 2), Y0 + BH + 20, "5 state slots", 12, 600, BODY, "middle") +text(cx(X0, 8), Y0 + BH + 20, "paged KV region, 4 of 7 chunks free", 12, 600, BODY, "middle") +rx = 612 +rect(rx, Y0, BW, BH, 5, "#ffffff", ACC_INK, 2, "5 4") +text(rx + BW / 2, Y0 + BH / 2 + 5, "#6", 13, 700, ACC_INK, "middle") +text(rx + BW + 14, Y0 + 15, "waits: every slot is taken", 13, 700, ACC_INK) +text(rx + BW + 14, Y0 + 33, "while the KV region sits half empty", 12.5, 400, BODY) + +el(f'') + +# kern: virtual ranges reserved once, one pool of physical chunks under them +SX, PX, VY, CY = 40, 428, 220, 324 +text(SX, 160, "KERN · VIRTUAL RANGES RESERVED ONCE, ONE POOL OF CHUNKS UNDER THEM", 11.5, 700, MUTED, spacing="1.2") +text(SX, VY - 34, "slots: per-sequence state", 12.5, 600, INK) +text(cx(SX, 5) - BW / 2, VY - 12, "#6 and #7 run", 12.5, 700, ACC_INK) +text(PX, VY - 34, "pages: paged KV", 12.5, 600, INK) +for i in range(8): slot(cx(SX, i), VY, str(i + 1) if i < 7 else "", "held" if i < 5 else "new" if i < 7 else "void") +for i, k in enumerate(["gone"] * 2 + ["free"] * 2 + ["held"] * 3 + ["void"] * 4): pages(cx(PX, i), VY, k) +text(cx(PX, 9), VY + BH + 18, "unmapped address space", 12, 400, MUTED, "middle") +text(cx(PX, 0) - BW / 2, VY - 12, "was 2 free pages", 12.5, 700, ACC_INK) + +tops, bots = VY + BH + 2, CY - 2 +for i in range(5): line(f"M{cx(SX, i)},{bots} V{tops}", LINE, 1.5) +for i in range(5): line(f"M{cx(SX, 7 + i)},{bots} L{cx(PX, 2 + i)},{tops}", LINE, 1.5) +for i in range(2): + line(f"M{cx(SX, 5 + i)},{bots} L{cx(PX, i)},{tops}", ACC_FILL, 2, "5 4") + line(f"M{cx(SX, 5 + i)},{bots} V{tops}", ACC, 2.5) +for i in range(12): chunk(cx(SX, i), CY, i in (5, 6)) +lx = cx(SX, 11) + BW / 2 + 18 +text(lx, CY + 11, "physical chunks", 12.5, 600, INK) +text(lx, CY + 28, "one pool, up to 64 MiB each", 12, 400, BODY) +bx0, bx1 = cx(SX, 5) - BW / 2, cx(SX, 6) + BW / 2 +line(f"M{bx0},{CY + 34} V{CY + 40} H{bx1} V{CY + 34}", ACC, 2) +text(cx(SX, 5) + P / 2, CY + 60, "remap: no bytes copied, off the serving thread", 13, 700, ACC_INK, "middle") + +# legend and the takeaway +LY = 410 +el(f'') +def key(x, draw, label): + draw(x) + text(x + 16, LY + 5, label, 12, 400, BODY) +key(40, lambda x: rect(x, LY - 7, 12, 12, 2, DARK, DARK), "held slot") +key(128, lambda x: rect(x, LY - 7, 12, 12, 2, ACC, ACC), "slot made by a remap") +key(285, lambda x: rect(x, LY - 7, 12, 12, 2, LINE, LINE, 1), "held page") +key(375, lambda x: rect(x, LY - 7, 12, 12, 2, "#ffffff", LINE, 1), "free page") +key(462, lambda x: rect(x, LY - 7, 12, 12, 2, "none", FAINT, 1.5, "3 2"), "unmapped") +key(554, lambda x: rect(x, LY - 7, 12, 12, 2, "#f1f5f9", LINE, 1.5), "chunk") +key(624, lambda x: line(f"M{x},{LY - 1} H{x + 12}", LINE, 1.5), "backs") +key(692, lambda x: line(f"M{x},{LY - 1} H{x + 12}", ACC_FILL, 2, "4 3"), "backed before the remap") +text(40, LY + 40, "One budget, no fixed line: pages and slots trade memory on demand.", 14, 700, INK) + +el('') +(Path(__file__).parent / "pool_budget.svg").write_text("\n".join(out) + "\n") diff --git a/docs/figures/pool_budget.svg b/docs/figures/pool_budget.svg new file mode 100644 index 0000000..2e43b7c --- /dev/null +++ b/docs/figures/pool_budget.svg @@ -0,0 +1,157 @@ + + +A STATIC SPLIT · FIXED AT STARTUP + +1 + +2 + +3 + +4 + +5 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +5 state slots +paged KV region, 4 of 7 chunks free + +#6 +waits: every slot is taken +while the KV region sits half empty + +KERN · VIRTUAL RANGES RESERVED ONCE, ONE POOL OF CHUNKS UNDER THEM +slots: per-sequence state +#6 and #7 run +pages: paged KV + +1 + +2 + +3 + +4 + +5 + +6 + +7 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +unmapped address space +was 2 free pages + + + + + + + + + + + + + + + + + + + + + + + + + + +physical chunks +one pool, up to 64 MiB each + +remap: no bytes copied, off the serving thread + + +held slot + +slot made by a remap + +held page + +free page + +unmapped + +chunk + +backs + +backed before the remap +One budget, no fixed line: pages and slots trade memory on demand. + diff --git a/docs/manifest.md b/docs/manifest.md index c3618bb..51f40b2 100644 --- a/docs/manifest.md +++ b/docs/manifest.md @@ -542,3 +542,17 @@ const,也不是 shape 意义上的 dim)、`kernels`→`ops`、`steps`→`lau kern test 的 c10 假阴性就是它咬的);`Program` 单字段包装拆掉;launch 的 `params`/`args` 有缺省;`extern:` 不写几何;ABI 常量折进 impl。删掉从没被读过的字段:`states.*.align`、`symbols.*.min`、scratch 实参的 `offset`、`u32` 标量。 + +## 命名常量(`constants`) + +可选的 `constants` 表给模型维度起可读的名字,用 C 风格的 `UPPER_SNAKE_CASE`: + +```json +"constants": {"VOCAB_SIZE": 151936, "HIDDEN_SIZE": 2560} +``` + +写 `"shape": ["seqs", "VOCAB_SIZE"]` 或 `{"i32": "HIDDEN_SIZE"}`。名字可以出现在任何数值字段里 +(`i64`、`f32`、表达式、容量、字节偏移),`schema_version` 除外。值只能是数值字面量,不支持别名和 +常量表达式;名字不能与 `vars` 重名。loader 在类型校验之前展开引用,所以目标字段的类型与范围照常 +检查;`buf` 这类纯字符串字段不受影响。序列化输出展开后的字面量。公开的 wire schema 由 +`kern_manifest::json_schema()` 从字面量 Rust 类型加这些引用分支生成。