Repository navigation
Conversation
… N", not a 500 (D5) Operator ruling D5 (ruling-3715-2010): "a 500 is a crash-class lie". A 27155-token turn on a dense CUDA model reached BorrowedCudaForward::reserve, failed with "the device KV cache holds 4096", and came back as HTTP 500 (h4-depcheck-0d, lambda + gx10 serve/code cells). - serving_context(): the model's context capped by the device KV cache; the borrowed serve session reports it, so max_tokens clamps to what is left. - AppState caches it at construction (PMAT-073: no read lock in the handler). - serve_context_refusal(): the pre-flight 400 in the CUDA chat, CUDA /v1/completions and /generate/stream routes, before any path takes the model, so a streaming client gets a 400, not a mid-stream error. - The direct paths map ContextLimitExceeded through generation_error_status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#3715 is the pv SHACL release-readiness shape. D5 is its serve × consumer-context child: a turn longer than the device KV returned 500. The quorum judged this diff against #3715's acceptance and failed it 3/3 on the mismatch, so the serve fix now has its own ticket and acceptance. Refs #4616, #3715 Agent: aprender-2e-serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, DEGRADED, advisory finding) Non-author reviewer (Sonnet 5), per COP 22:16Z applying the 21:10Z committed- .minisig route. diff_patch_id cf52ad666e. Verdict DEGRADED: cuda (docs MCP OAuth-gated) and mutation (no cargo test on lambda) unreachable; pmat and agy gemini-3.1-pro-high consulted; crux not triggered. Guard rejects only B1 (unsigned) until the CI signer's .minisig is committed. Advisory finding (measured): /generate and /batch/generate still map every CUDA error to 500 (batch_processing.rs:492, batch.rs:515). Follow-up, not folded here: changing the diff would void this receipt. Agent: aprender-2e-serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…receipt Per COP 22:16Z (21:10Z committed-.minisig route). The .minisig is verbatim from pr-review-signature check-run 109170961010 (CI signer, repository key). minisign -V under .github/pr-review.pub: verified, trusted comment pr=4614 pid=cf52ad666e. check_pr_review_receipt.sh: ACCEPT (positive controls fired). Agent: aprender-2e-serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g the serial prefill (#3715) The #3715 lambda cells timed out or failed at the 20k rung on two causes, both measured on the RTX 4090 with apr 0.70.0 (8d021f6): - Qwen3-1.7B (QK-norm): `[PREFILL-PATH] serial cc=89 (qk-norm model)`. One forward_gpu_resident per prompt token: 7955 tokens took 109 s, 26131 did not finish in 300 s at GPU util 99-100 %. Not #4609: the backend was the GPU throughout. - Qwen2.5-0.5B IQ3_M (batched): CUDA_ERROR_OUT_OF_MEMORY in prefill_all_layers_gpu at 26148 tokens. One pass allocates a num_heads x S x S f32 score matrix (38 GB), and its u32 row offsets wrap past 2^32. The batched prefill now runs in chunks whose score matrix fits the 1 GiB budget the Qwen3.5 prefill already uses; each chunk appends to the KV cache through the existing cache_len > 0 attention path. A prompt that fits one chunk runs exactly as before. QK-norm models keep FP8 off and now take the FP16 batched prefill, which #3413 measured passing CPU parity; serial was chosen only because the unchunked prefill OOMed 8B. Refs: #3715 Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…3715) apr chat built its CUDA model with a flat 2048-position KV. The dense session refuses a longer turn on the device, so a 26k-token chat turn on Qwen3-1.7B ran on the CPU and hit the 600 s cell timeout (GPU util 0%, measured on lambda). apr run already sizes its KV to the turn (#4268); chat loads before it has read a turn, so it now takes the model context, capped by free VRAM after the weights, the FP16/FP8 prefill cache, the GH-178 reserve and the chunked-prefill score budget. When nothing is left (Qwen3-8B on 24 GB), it keeps the 2048 floor. Refs: #3715 Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…3715) The Qwen3.5 batched prefill dequantizes each weight to F32 for one SGEMM, and had kernels for Q4_K/Q5_K/Q6_K/Q8_0 only. Qwen3.5-4B-UD-Q4_K_XL (F16 ssm_alpha/beta, IQ4_XS FFN blocks) and Qwen3.5-0.8B-IQ4_XS therefore refused the CUDA prefill, the F2 guard rejected the GPU, and every lambda cell ran on the CPU (GPU 0%) into the 600 s timeout. Adds F16DequantKernel and Iq4XsDequantKernel. The values match the per-token GEMV weights bit for bit (device tests against the ggml reader). Refs #3715 Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ched prefill does not fit (#3715) 3c2a354 moved QK-norm models from the serial prefill to the chunked FP16 batched one. The FP16 weight cache is warmed on either path, but only batched also needs the prefill workspace. Qwen3-8B on the 24 GB 4090 has 4.7 GB of weights and a 15.1 GB FP16 cache, which leaves 0.1 GB, so init_prefill_workspace failed and a 2k chat turn fell back to the CPU (382 s, rg5 C-q38b-chat-2k). The path is now decided against the device before the cache is warmed: weights + cache + KV + prefill reserve over free VRAM keeps serial, as on the base. Qwen3-1.7B (about 9 GB of 23) stays batched. BATCHED_PREFILL still overrides. The resident-bytes estimate is shared with session_kv_len. Refs: #3715 Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mutation unreachable) Reviewer agent:claude-sonnet-5/pr-review-4620; author claude-opus-5-5/aprender-ec. Written without diff_patch_id or .minisig. The CI signer stamps both. Refs: #3715 Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…they called (#4621) The last loop in fit_check_leaves_non_qk_norm_fp8_and_forced_paths_alone built two fresh profiles and asserted they were batched, so a mutant that switched the path to Serial while still returning false passed it. Each case now asserts the path of the profile it called. Red on that mutant, green on the code (intel). Quorum finding, PMAT-4621 round 1. Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uda_backend no longer grows x86-main on 099d07d was red on this diff, not on the runner: - complexity: try_cuda_backend grew (cog 43->46, cyc 14->15). The chat pre-flight now rides the tokenize result through fit_serving_context, so the function gains no branch. - mutants: serving_context lived in the cuda-gated dense_session_borrowed, where no CPU lib test can reach it. The rule is now dense_session::cap_context(context_length, device_kv), CUDA-free, with lib tests on every arm; fit_serving_context has lib tests for pass, refuse (400) and a kept tokenize error. - roadmap.yaml regenerated for the PMAT-4616 fragment. - surface_audit.csv: the /v1/completions row's cited line moved 398 -> 405. Refs #4616 Agent: aprender-2e-serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mutation sweep unreachable; targeted mutant killed) Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… tree
Its findings.sarif carries a placeholder excerpt_sha256 ("n/a-short-code-idiom-not-hashed")
that S1.1 would reject once signed, and the receipt binds that sarif by sha256, so the
author cannot correct it without editing the reviewer's record. It is superseded by the
65370ad receipt (the head's code) and stays in history at 18b6c83. Quorum finding,
PMAT-4621 round 2 (gemini lane).
Agent: aprender-ec
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…igh, haiku-4-5) at 2b8d79a Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… 4 computes (b3f1fd942a) Arm 4 A2 reads predicate.diff_patch_id from the COMMITTED receipt; the CI signer stamps only its runner copy, so an unstamped receipt is legacy and RED. Computed with scripts/lib/pr_review_patch_id.sh prpid_compute over 722fd16..HEAD (evidence and quorum json excluded), equal to the value Arm 4 printed in run 36510383864. Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ceeded to 400 Quorum lane 2 (gemini-3.1-pro-high) at 926edb2 found two GPU-resident generate paths that still hardcoded 500 for every generation error, bypassing generation_error_status(): batch_processing.rs try_cuda_generate and batch.rs try_cuda_batch_generate. Both now route the status through it, so an over-length request is a 400 on the CUDA path as on the CPU path. cuda clippy on intel: only the pre-existing residual.rs:314/343 borrow_deref_ref. Agent: aprender-2e-serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent: aprender-2e-serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he signature arm Arm 4 (run 36511265641) verified the signature, then rejected the content: the antigravity divergence counted 1 against 0 findings. The reviewer corrected it to the agy output it actually recorded (findings: []), added the pmat duplication fields the guard requires, renamed two SARIF drivers to the allowed set, and re-bound findings_ref. The guard ACCEPTs it under a throwaway key. diff_patch_id is unchanged. Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… through generation_error_status Quorum at 1ead14b (sonnet-5 + gemini-3.1-pro-high FAIL) found more handlers that hardcoded 500 on a generation error, so ContextLimitExceeded still leaked as 500: SafeTensors CUDA chat, GPU /generate, logprobs, perplexity, the APR/demo batch + stream paths and the openai GPU fallback. Swept all 12 generate-error sites in api/ (decode errors stay 500: those are server faults). generation_error_status keeps every non-context error at 500, so only the over-length case changes. intel: clippy --lib default rc0; --features cuda only the pre-existing residual.rs:314/343 borrow_deref_ref; 15 targeted lib tests pass. Local mutants not_measured (unmutated baseline timed out at 600s). Agent: aprender-2e-serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…6/IQ4_XS device dequant) The diff held 97 mutants against the gate's cap of 60. The QK-norm fit check (root 2) and the chat session KV (root 3) move to their own PR, per the cop's split ruling; they are restored here to the merge base, unchanged in content. Refs #3715 Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (root 2) Qwen3-family models were forced onto the serial prefill by disable_fp8_for_qk_norm (#3413/#3483); serial scales super-linearly and the 1.7B 20k cell ran past 600 s. They now keep FP8 off but take the FP16 batched prefill, which #4620 made chunked. A model whose weights + FP16 cache + KV + prefill reserve exceed free VRAM keeps serial (Qwen3-8B on 24 GB, which OOMed into a CPU fallback without this check); BATCHED_PREFILL=1 still overrides. The resident estimate walks the projection tensors themselves (bytes and in x out elements) instead of a dims formula, so it cannot drift from the model. Stacked on #4620. Refs #3715 Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
apr chat built its CUDA model with OwnedQuantizedModelCuda::new, a flat 2048 positions, so a 26k-token turn was refused on the device, ran on the CPU and hit the 600 s cell timeout. for_session gives the model's whole context when free VRAM holds it beside the weights, the prefill cache and the reserve, else the old 2048 floor. Stacked on the root-2 PR. Refs #3715 Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, DEGRADED, advisory findings) Non-author reviewer (Sonnet 5). Verdict DEGRADED: cuda (docs MCP OAuth-gated) and mutation (no cargo test on lambda) unreachable. Confirms the 167bd5e receipt's finding is fixed (try_cuda_generate / try_cuda_batch_generate now route through generation_error_status). Guard rejects only B1 (unsigned) until the CI signer's .minisig is committed. Advisory follow-ups, not folded (changing the diff voids this receipt): batch_processing.rs:343 batch_generate_gpu still 500; /generate and /batch/generate have no serve_context_refusal pre-flight; the effective_max_tokens `>` vs serve_context_refusal `>=` boundary. Agent: aprender-2e-serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…7e2f The reviewer (sonnet-5) had left a placeholder in predicate.diff_patch_id; it now carries the pinned prpid value, equal to the one CI's Arm 4 measured (base 2817c6d). Guard: only B1 (unsigned) until the signer's .minisig. Agent: aprender-2e-serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…guard-complete The signature exposed a B1 the unsigned state had hidden: crux was `consulted` without surfaces/comparative_claims. Reviewer (sonnet-5) filled the S3.C arrays honestly and added the missing citation fields; verified ACCEPT under a throwaway key. patch-id binding unchanged (2fe3407e2f). Agent: aprender-2e-serve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… graph PREFILL_GRAPH=1 (opt-in, off by default) replays a graph that bakes positions 0..S-1 and resets the KV lengths to 0. With the chunked dense prefill, chunk 2+ of the same length would have replayed it over a filled KV cache, silently corrupting it. Only a chunk starting at position 0 may take the graph now; the rest run eager. Found by the non-author Arm-4 review of #4620. Refs #4621, Refs #3715 Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s 2+3, now in the stacked PRs) Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-4621) Non-author Sonnet 5 review of part A (chunked dense prefill, F16/IQ4_XS device dequant, PREFILL_GRAPH first-chunk-only). Verdict DEGRADED; diff_patch_id stamped. CI signer adds the .minisig. Refs #4621 Agent: aprender-ec Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto B Agent: aprender-e6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent: aprender-6b Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, editing none of its lines R2: #4623's hunks stay as they are; K1 only adds the FitServing arm and its sizer. Agent: aprender-00 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#4603) A cuda-only aprender-serve lib test, so yoga's cuda-unit job runs it with no model file. An 8192-context llama-shaped model (above serve's old fixed 4096) is sized by `for_serving` and served through the real router and the continuous-batching scheduler: - the device KV is 8192, the model's context; - /v1/completions with 8188 prompt tokens + max_tokens 4 answers 200 with all 4 tokens (usage 8188/4, finish "length"); - 8192 and 8193 prompt tokens answer 400 "context exceeds 8192 ... refused whole, not truncated", not a 500 and not a truncated 200. The weights are centred Q4_K with an FFN norm and separate Q/K/V. The shared fixture's all-positive d=1.0 pattern (cosine 0.00) and its missing FFN norm (cosine 0.54) both fail the CPU/GPU parity gate, correctly. Without a device it prints "CUDA model init failed", which cuda-unit counts as a failure, not a pass. Agent: aprender-00 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…with a 400 (K1 #4603, D5 #4614) KV-only vehicle for #4630, folded (moved, not copied) from the ec stack onto main. Three rows: - #4603 K1: `apr serve` builds the CUDA model with `for_serving`, which sizes the device KV to the model's context capped by free VRAM (two batched slots budgeted, never below the old 4096). Before, serve fixed it at 4096 whatever the model declared, so a longer prompt overflowed the cache and answered 500. - #4614 / PMAT-4616 D5: a prompt at or past the serving context is refused whole with 400 "context exceeds N", on chat, completions and stream. - The KV budget core it needs (session_kv.rs, resident.rs). C25 test: api/tests/serve_kv_limit_k1.rs builds a synthetic 8192-context model, serves it through the real router and batch scheduler on the device and asserts KV == 8192, 8188+4 -> 200 with all 4 tokens, 8192 and 8193 -> 400 "refused whole". A return to Fixed(4096) fails the KV assertion. gx10 (cc 121): aprender-serve --features cuda lib filter serve_kv_limit_k1|serve_context_d5|session_kv|resident 28/28, apr-cli serve 429/429. Refs #4603 #4614 #4630 Agent: aprender-00 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…op 11:03Z) The tree is exactly devont/k1-kv-only (origin/main + K1 #4603, D5 #4614 and the KV budget core). The other stack rows (#3715 serial prefill / for_session, #3413 message, #4620, #4623, #4624) stay on their ec parents for a later PR; nothing is deleted there. A merge, not a rebase or force-push. Agent: aprender-00 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
Agent: aprender-00 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#4630 roadmap.yaml conflict: both sides appended an entry (PMAT-4616 here, PMAT-4633 on main); kept both, ids unique (check_roadmap_ids_unique PASS). Agent: aprender-kv Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rving context N #4630 carries K1 and D5, but only D5 had a ticket, so the Q0 r1 gemini lane judged the K1 KV sizing out of scope and read AC1's literal "exceeds 4096" as forbidding it. K1 now has its own ticket, and AC1 says N is the serving context (4096 before K1, the model-sized KV with it). Agent: aprender-kv Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Q0) gemini-3.1-pro-high (agy) PASS + claude-sonnet-5 PASS + claude-haiku-4-5 PASS, author claude-opus-5-5, not degraded. r1 at 6a6a1c2 was NOT AGREED: the gemini lane judged K1 out of scope because it had no ticket (fixed by PMAT-4636). Advisory finding: /v1/batch/completions has no pre-flight refusal; it is not one of the three surfaces the tickets name. Agent: aprender-kv Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…et reviewer) Receipt + SARIF from an independent reviewer invocation (agent:claude-sonnet-5-5/pr-4630-review); author aprender-kv (opus-5-5). DEGRADED: mutation not measured (no cargo on lambda), K1 device claims unmeasured, cuda-docs-reviewer not dispatched. Unsigned: CI signs and stamps diff_patch_id. Pmat-Ticket: PMAT-4616 Agent: aprender-kv Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts resolved to main's side: prefill.rs (main adds the #4621 refusal tests), the PMAT-4621 quorum audit and the #4620 pr-review evidence (#4620 has landed). The PR's remaining diff vs main is the chat-session KV work in 5 files. Refs #4621 Agent: aprender-78 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noahgift
enabled auto-merge
October 4, 2026 08:52
First parent main, second the PR tip: the push is a fast-forward. D5 (PMAT-4616) landed on main via #4655, so serve_context_d5.rs and the PMAT-4616 quorum/roadmap entries keep main's copy. K1 stays: serving_cuda_model(), session_kv.rs, resident.rs, serve_kv_limit_k1.rs, PMAT-4636. handler_gpu_completion.rs keeps #4254's model_source measurement before the move into CUDA. roadmap.yaml regenerated (make roadmap-aggregate-check idempotent). Agent: aprender-6a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verdict DEGRADED: the mutation arm could not run on the build host. pmat, cuda and the cross-vendor arm were consulted; crux not triggered. Unsigned by design: the CI signer stamps the patch id and adds the signature. Agent: aprender-78 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… beside chat's session KV Merges the #4630 branch (devont/k1-serve-kv @ da4f4f9) into this one by merge, so its commits and its review receipt keep their history. The two branches share a merge base that already holds this PR's session-KV work, and #4630 later dropped that work to stand alone on main. A plain merge reads that drop as a revert: it was clean, and it silently took gpu_profile.rs and chat_load_tokenizers.rs back to main and removed for_session from gguf/cuda/mod.rs. This merge keeps both PRs' changes against main instead: - gpu_profile.rs, chat_load_tokenizers.rs: this PR's version (#4630 leaves them as on main). - gguf/cuda/session_kv.rs: #4630's version, which holds every line of this PR's. - gguf/cuda/mod.rs: both KV modes side by side -- FitContext with for_session and session_kv_len (#4621), FitServing with for_serving and serving_kv_len (K1), both match arms, and the QK-norm prefill fit check (#3715). Against either branch, the result drops no line that branch changed from main. Refs #4603 #4614 #4616 #4621 Agent: aprender-wprodmodels Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Verdict DEGRADED, no blocking finding: cargo and the mutation arm were not run on the review host. The union of #4623 and #4630 holds: every line #4623 changed is kept, and the 2 lines of #4630 not kept are main's text that #4623 edited. apr chat reaches for_session, apr serve reaches for_serving, and the QK-norm fit check runs in both modes. One advisory: under FitServing the fit check counts one KV slot; fit_batched_kv_alloc shrinks the batch slots, so it degrades gracefully. Unsigned by design: the CI signer stamps the patch id and adds the signature. Agent: aprender-wprodmodels Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
noahgift
added a commit
that referenced
this pull request
Oct 5, 2026
The fold put 78 CUDA-gated mutants (gguf/cuda/mod.rs 41, cuda/gpu_profile.rs 17, gguf/cuda/resident.rs 12, gguf/cuda/session_kv.rs 8) in this diff, over the mutants-cuda cap of 60 (CI run 37374575186), and gate needs that section. Raising the cap would loosen a gate, so the fold comes back out instead: this reverts the merge of ec/4621-chat-session-kv (b777d5c) and the two K1 test commits on top of it (68b2819, e95438e). #4711 keeps only the tools/template fix. #4623 lands separately, split under the cap, and can be re-folded after it lands. Agent: aprender-w4711 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
served() printed a skip line and returned None, and the test returned
early. libtest counts that as a pass. cuda-unit's skip grep caught the
line on yoga, but any other `--features cuda` run without a device
reported K1 green having measured nothing.
served() now panics on a CUDA init failure ("K1 not measured: ...") and
returns the router directly. The test keeps no #[ignore]: cuda-unit
runs the cuda-only lib tests by name without --ignored, so an ignored K1
would leave CI and a return to a fixed 4096 KV would pass.
Agent: aprender-wprodmodels
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noahgift
disabled auto-merge
October 6, 2026 03:05
Verdict DEGRADED, no blocking finding. Mutation could not run (no cargo on the review host) and K1 was not run by the reviewer. The K1 change holds: no skip, no early return, no #[ignore]; cuda-unit still runs it by name on a device; no CI job runs the serve cuda lib tests on a device-less runner. One new advisory: --all-features make targets now fail on K1 on a device-less host. Agent: aprender-wprodmodels Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
Closed by the PR-cap pass (open PRs stay at 10 or fewer). This PR has been red or in conflict with Nothing here is rejected, and the branch is kept. Reopen this PR when a slot opens after the first change of 0.71 lands (it removes the mutants and provable-ladder jobs from PR CI). Then merge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
D3-SPLIT part C of #4620 (root 3). Stacked: A (#4620) → B (
ec/4621-fit-and-chat-kv, owner aprender-e6) → C (this PR). C needs B'sresident_estimate.What
apr chatbuilt its CUDA model withOwnedQuantizedModelCuda::new, a flat 2048-position device KV. A 26k-token turn was refused on the device, ran on the CPU and hit the 600 s cell timeout (GPU util 0%).for_sessionsizes the device KV to the model's context when free VRAM holds it beside the weights, the prefill cache and the reserve, else keeps the old 2048 floor.gguf/cuda/mod.rs:KvLen { Fixed, FitContext },for_session,session_kv_len(glue over B'sresident_estimate)gguf/cuda/session_kv.rs(new): the pure length rule + testsapr-clichat_load_tokenizers.rs: GGUF and APR chat CUDA init callfor_sessionBase / CI
Base is B's branch, so CI (which runs only for PRs into
main) does not run yet. When B merges, this PR is retargeted tomain; its diff vs main is then this one commit. Quorum (--ticket PMAT-4621) and the Arm-4 receipt are produced on that head, and the cop is asked before arming.Receipts
Mutants on this diff: 12 (≤60 cap). Local receipt: see the first comment.
Folded in: #4630
#4630 (K1:
apr servesizes its GPU KV to the model context, and a request past it is a 400) is folded in by merge at 616ff4c; its branch is kept. The two branches share a merge base that holds this PR's work, which #4630 later removed, so git's clean merge would have reverted this PR's chat session KV. The fold resolves to the union of both diffs against main instead:KvLen::FitContext/for_session(chat) andKvLen::FitServing/for_serving(serve) side by side, with the QK-norm prefill fit check in both. Review receipt:evidence/pr-review/4623/3cc3d7a71b720d29b21fa3beb4fdf11c837700ea/.Fixes #4621
Refs #3715 #4603 #4614
🤖 Generated with Claude Code
keep-open: #3715 is the root issue, closed by part A (#4620); this PR is part C of the split.