K1: apr serve sizes the GPU KV from the model's context within free VRAM, not a fixed 4096

Child of #3715 (ruling-3715 K1). Carried by PR #4630 together with D5 (#4614 / PMAT-4616).

Before: apr serve fixed the device KV at 4096 whatever the model declared, so a prompt past 4096 tokens on a longer-context model overflowed the KV and came back as a 500.

Acceptance:

apr serve builds the CUDA model with OwnedQuantizedModelCuda::for_serving; with no --context-length the device KV = serving_kv_len = min(model context, what free VRAM holds with SERVE_BATCH_SLOTS=2 batched slots budgeted), floored at 4096 (or the model's context when shorter).
An explicit --context-length (≠ default) still sizes the KV via REALIZR_CONTEXT_LENGTH.
The D5 refusal limit (PMAT-4616) is the serving context this KV yields: at the limit 200 whole, one past it 400, never 500. api/tests/serve_kv_limit_k1.rs checks this on the device with an 8192-context model (above the old 4096, so a return to the fixed size fails).
session_kv.rs unit tests: small model gets its whole context; a model that fills the card keeps the 4096 floor; partial room is used and never exceeded.

Agent: aprender-kv

docs(roadmap): PMAT-4636 K1 entry (#4636); PMAT-4616 AC1 names the se…

Status (aprender-clerk, owner): not on main. At cc71bde, main has no for_serving, serving_kv_len, SERVE_BATCH_SLOTS, session_kv.rs or serve_kv_limit_k1.rs.

fix(serve): K1 — GPU KV = model context within VRAM, not a fixed 4096 (#4603) #4630 (K1 + D5) was folded into fix(#4621): apr chat sizes its device KV to the context (D3-SPLIT C, root 3) #4623 at 616ff4c. fix(#4621): apr chat sizes its device KV to the context (D3-SPLIT C, root 3) #4623 was then closed by the PR-cap pass on 10-07 ("nothing here is rejected, branch kept; reopen when a slot opens after the first change of 0.71 lands").
The kept fix(#4621): apr chat sizes its device KV to the context (D3-SPLIT C, root 3) #4623 branch (688a0e2) is 131 commits behind main and 53 ahead. A test merge onto main gives 0 conflict markers. Three files changed on both sides: serve/handler_gpu_completion.rs, api/tests/mod.rs and roadmap.yaml; the roadmap is regenerated by the aggregator.

Next: re-cut onto current main as one K1 PR, after cargo test -p aprender-serve --lib (session_kv unit tests) passes on a CPU host. The acceptance test serve_kv_limit_k1.rs needs a CUDA device with an 8192-context model, so it needs a GPU host slot. The PR opens on a release-train slot.

🤖 Generated with Claude Code

P0 to P1 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, so it stays on the 0.71.0 milestone at P1.

Folded into #3715 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.

Agent: aprender-clerk

Navigation Menu

Repository navigation

K1: apr serve sizes the GPU KV from the model's context within free VRAM, not a fixed 4096 #4636

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

K1: apr serve sizes the GPU KV from the model's context within free VRAM, not a fixed 4096 #4636

Description

Activity

noahgift commented on Oct 12, 2026

noahgift commented on Oct 14, 2026

noahgift commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions