Repository navigation
feat(serve): logprobs on the chat route, and the V3 serving-shape probe that reads them (PMAT-4971) - #5188
Open
noahgift wants to merge 34 commits into
Open
feat(serve): logprobs on the chat route, and the V3 serving-shape probe that reads them (PMAT-4971)#5188noahgift wants to merge 34 commits into
noahgift wants to merge 34 commits into
Conversation
…in (PMAT-4971) V3-a step 1. `top_k_logprobs(logits, k)` returns the k best tokens with their logit and logprob, best first, ties broken by the lower id. The top-1 id is the argmax the engine already picks, and the margin V3 reads is top[0].logit - top[1].logit. NaN is never ranked; a 1e30 logit stays finite. Only `gguf/logprobs.rs` comes over; #4026's CLI and session hunks do not. Nothing calls it yet: the request fields, the engine hook and the wire shape are the next commits on this branch. (cherry picked from eaab36b, #4026) Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 63282de) (cherry picked from commit 3c7e778) (cherry picked from commit f4e9278)
…s the device argmax (PMAT-4971) Session::generate_with_logprobs records, per generated token, the chosen token's logprob and the top-N tokens with their raw logits, from the logits the choice read (after the repetition penalty, before temperature). generate delegates to it with no recording, so its behaviour is unchanged. A device-greedy step keeps its device argmax. forward_greedy takes read_logits; when it is set the CUDA path copies the logits that argmax read to the host after it, and the token never comes from the host: - decode: forward_gpu_resident_to_token_id_reading (new gguf/cuda/token_logits.rs) takes the same one of the three decode paths with the same argmax; the replay path copies the workspace logits buffer (download_workspace_logits), the CPU-argmax and eager/capture paths hand over the logits they downloaded. - prefill: FirstToken::TokenAndLogits runs the PMAT-083 extraction steps on one logits buffer (new cuda/executor/layers/first_token_logits.rs) and copies it after the GPU argmax, so token and logits come from the same buffer. A device path that was asked for the logits and returned none is an error, never a silent host recompute. uses.rs and prefill.rs are untouched: both carry older functions over the pre-commit complexity cap, so the new paths live in their own modules. Contract apr-serve-openai-compat-v1 1.21.0, ASOC-INV-021. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 64c7649) (cherry picked from commit b6560e7) (cherry picked from commit 3a9bcb7)
…kend refuses 501 (PMAT-4971) V3-a slice 2 of #4971 (ASOC-INV-021). /v1/chat/completions now reads `logprobs` and `top_logprobs`. Before this, serde dropped both keys, so a client that asked for logprobs got a 200 without them. - `top_logprobs` outside 0..=20 is refused when the request is deserialized, and the error names the field. `top_logprobs` without `logprobs: true` is a 400 from `field_conflict`, next to the thinking toggle. - The quantized CPU non-streaming turn runs `dense_turn_with_logprobs`. It keeps one record per generated token and drops the stop token's record. A count check refuses a reply whose entries do not match its tokens. - Logprobs are inserted into the serialized reply, so a request without them gets the same bytes as before and no `logprobs` key. - Every backend without a logprobs path answers 501 and names itself: GPU, cached, CUDA, SafeTensors CUDA, APR f32, APR Q4K, Qwen3-MoE, Qwen3.5, the dense registry, and the quantized stream and traced loop. Streamed entries are a later slice. - `BPETokenizer::decode_bytes` gives a token's exact bytes. `decode` was at cognitive 41 and is split into helpers with the same behaviour. FALSIFY-LOGPROBS-WIRE-4971 (`api::tests::logprobs_4971`, through the real router) now states the 501 for a streamed request. It compares an entry with top_logprobs[0] by logprob, because the fixture's logits can tie. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 7976551) (cherry picked from commit 230031c) (cherry picked from commit 490d395)
…971)
Slice 3 of V3-a. A quantized `stream: true` request for logprobs was
refused 501 ("quantized (streaming)"); it is now served.
- dense_stream_with_logprobs hands each token on with its step record.
The stop token's record is dropped with the token, and a token without
a record ends the turn with an error. dense_turn_with_logprobs is now
built on it.
- The engine sends each record ahead of its token, on an unbounded side
channel (StreamLogprobs). The stream holds each entry until a chunk
carries text, and that chunk carries all held entries. Whatever is
still held at the end goes on the terminal chunk. A token whose record
is missing, or belongs to another token, ends the stream with an error
event.
- Entries are inserted at the JSON level, so a stream without logprobs
serializes exactly as before.
- spawn_quantized_stream takes the stream out of try_quantized_backend.
Its cognitive complexity drops from 25 to 21.
- Only the traced loop still refuses ("quantized (traced)").
The tests check four things:
- The streamed entries sum to usage.completion_tokens.
- They equal the unstreamed reply's entries, token for token.
- No chunk has a logprobs key without the request.
- A traced request is refused 501 by name.
FALSIFY-LOGPROBS-WIRE-4971 is updated to match.
Refs #4971
Agent: la-0.71.1a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b086a4b)
(cherry picked from commit bebdd5b)
(cherry picked from commit 9b12726)
…MAT-4971) The hybrid refused `logprobs` with 501. Its session is the same `Session` the dense CPU path runs, and `generate_with_logprobs` already records each step there, so the backend now serves them instead: - one `qwen35_generate` takes the lock, keeps the stop token (and its record) from the callback and stores `on_gpu`, for both wires; - the unstreamed reply drops the stop token in `qwen35_turn` and refuses a reply whose records do not number its tokens; - the stream sends each record ahead of its token to `live_sse_response`. The hybrid has no device argmax, so a step's record is read from the logits its token was chosen from, on CPU and GPU alike. Without `logprobs` the turn is `generate` as before (`generate` is `generate_with_logprobs` with no `top_n`). New falsifier FALSIFY-LOGPROBS-QWEN35-4971 (CPU, stream, and GPU under --features cuda) on the Qwen3.5-0.8B fixture; an absent model is a SKIP line, which is not_measured. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 1bdda28) (cherry picked from commit 4ebaa9e) (cherry picked from commit 746b32e)
… test init gets its new fields (PMAT-4971) mod_create_demo.rs is include!d into api/mod.rs, so `super::` reached the crate root, where TopLogprobs is not. The new `reads`/`read_back` fields of the DeviceGreedy test engine were missing from one initializer. Both broke the aprender-serve lib and lib-test builds on lane S (ac93037). Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 042d751) (cherry picked from commit cff2c26) (cherry picked from commit 35eb553)
…_batch into named steps (PMAT-4971) No behaviour change. cuda_batch_scheduler_loop (cognitive 85) and process_cuda_batch (cognitive 72) were each one function; the pre-commit hook fails any staged file holding a function over 25, so no change to this file could land. Each step is now its own function, same order, same lock scopes, same log lines: - loop: drain_adaptive (+ wait_for_stragglers), accumulate_window (Accumulated::Closed runs the last batch then stops, as before), log_batch_done. - batch: run_fast_path (+ ttft_trace), phase1_inputs, setup_and_prefill, decode_step (join_slot, recycle_done_slots), fail_all; the five copies of the per-token callback are token_callback. The one dropped line is `if !batch.is_empty()` before the channel-closed batch: the batch always holds the request that opened it. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b37642e) (cherry picked from commit b117664) (cherry picked from commit dfe1c9b)
…ugh the scheduler's m=1 path (PMAT-4971) The CUDA chat backend answered `logprobs: true` with 501. It now serves ASOC-INV-021 the way the quantized CPU and Qwen3.5 paths do: - The request carries a RecordSink (top_n + a record channel). The dense turn sends each step's record before its token, on the direct path (dense_cuda_turn) and on the batch scheduler's m=1 fast path (dense_stream_recording), streamed and not. - The handler takes a record per token with a non-blocking try_recv, so a missing record is an error chunk / 500, never a hang. A batched (m>1) turn records nothing yet, so it fails closed until the batched slice. - The traced CUDA loop records none: 501 naming `CUDA (traced)`. - try_cuda_backend's streamed branch moves to cuda_streamed_reply, which keeps it under the cognitive cap. FALSIFY-LOGPROBS-CUDA-4971 drives the real router on tinyllama, direct then scheduled: one entry per completion token, the same reply as without logprobs, the same entries streamed, and the traced 501. No device or no model prints SKIP, which is not_measured. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5872635) (cherry picked from commit 3716785) (cherry picked from commit 4cda1f8)
The batch scheduler's batched path recorded no logprobs, so a request that asked for them failed whenever it shared a batch. Each slot now carries an optional recorder. The decode step records a live slot's logprobs from the same logits its token came from: the sampled path from the logits it downloads, the device-argmax path from logits_buf, downloaded after the argmax so the token stays the device's. A recording step runs eager, since BATCHED_GRAPH replays into its own buffers. A record reaches the handler before its token; a stop token's record is dropped. A joined or recycled slot records nothing until the scheduler gives it its request's recorder. FALSIFY-LOGPROBS-CUDA-4971 gains a pair decoded as one m=2 batch, shown by the scheduler's in-flight peak: one slot with logprobs and one without (same text, entries only where asked), then both asking, one streamed (equal entries). Only the iteration scheduler still records none. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 943a091) (cherry picked from commit 294cf0d) (cherry picked from commit 2a70ee2)
…-4971) No behaviour change. The iteration scheduler's batch function (cognitive 101) becomes setup, a completion test, one iteration step, the slot scheduling it does under the lock (recycle done slots, else join a waiting request) and the closing of done slots, each under the hook's limits, so the next commit can give a batched slot its logprobs recorder. Its callbacks use the batch scheduler's token_callback, now pub(crate) with its type aliases. "!joined && !has_done" is "!has_done", since only a recycle joins. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 414cb14) (cherry picked from commit d43adca) (cherry picked from commit 698b2c8)
…(PMAT-4971) A request that asks for logprobs now records them in the iteration scheduler too. A slot gets its recorder whether its prompt was prefilled at setup, joined a free slot mid-batch, or was recycled into a finished slot; each is set only after the engine call succeeds, as the batch scheduler does. FALSIFY-LOGPROBS-CUDA-4971 drives process_iteration_batch directly, so the batched path is not a race: two prompts set up (one asking, one not) and two waiting, with slots for 3. Every turn runs to its max_tokens. Each turn that asked gets one record per token, steps from 0, naming the token sent, best 3 first, every logprob <= 0, the chosen one as likely as the best. Before this change those turns got no records, so the count check fails. The handler still fails closed on a token without its record; the docs no longer name the iteration scheduler as a turn that records none. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d83020f) (cherry picked from commit 76fa08c) (cherry picked from commit e5aa87b)
… name (PMAT-4971) ASOC-INV-021 forbids a 200 without logprobs to a request that asked for them. Four apr serve chat routes are not mounted by realizar's router and parse the body as untyped JSON (or, in banco, a type without the fields), so they dropped `logprobs` and answered 200: - APR CPU fallback (AprTransformer): 501 naming `APR CPU (AprTransformer)` - SafeTensors F32 CPU: 501 naming `SafeTensors F32 CPU`, before the no-transformer 503 - WGPU: 501 naming `WGPU` - banco: the request type gains `logprobs` and `top_logprobs` (typed, so 21 is refused when parsed); 501 naming banco All four answer as the router does: `top_logprobs` without `logprobs: true` is 400, an out-of-range `top_logprobs` or a non-bool `logprobs` is 422 naming the field. One check in realizar decides it for every route (api::json_logprobs_refusal for JSON bodies, logprobs_fields_refusal for typed ones); the router's own reasons now come from the same two text functions. Falsifiers: a case table on the check (api::chat_logprobs tests), FALSIFY-LOGPROBS-SERVERS-4971 (apr-cli tests_logprobs_4971, the real APR CPU and SafeTensors routers with no model) and FALSIFY-LOGPROBS-BANCO-4971 (banco's default router). WGPU has no router test because its state needs a GPU device; it is covered by the case table and a --features wgpu build. Not changed: build_gpu_router (handlers_include_01.rs) has no caller, and the server `apr compile` generates is out of scope. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 570d686) (cherry picked from commit 6e60f3e) (cherry picked from commit 5b1eda6)
…PP-26 witness (PMAT-4971)
V3-d. perf_gate::v3_shape ports the hand-run v3_shape_check.py from the
V3 spec branch: a perf041 witness is a serving-shape parity receipt only
for the blessed Qwen3.5-4B (prefix, sha256), with identity fields, bands
c in {1,4,8,16} each once with m_formed == c and PASS, and agreement with
m=1 to declared_min unless a recorded top-2 margin below 0.05 explains a
near-tie. Exit 0/1/2 as in the reference. It closes the draft contract's
findings: unreadable is RED (F1), typed fields (F2), no declared_min is
not admissible (F3), the reason names the margin and eps (F4), a band
with no divergence measurement reports its result only (F5), a duplicate
c is not admissible and every copy is checked (F6). The case table also
plants the two mutants the Python table let survive (prefix vs substring,
a defaulted declared_min).
BatchInvarianceWitness gains top2_margin_at_divergence (master §12 row 22),
None from token comparison and off the wire when None, so older receipts
are byte-identical; it drops Eq for the f64. apr test llm bench reads it
from the perf041 band and refuses a non-number by name.
Not a gate: nothing on a merge, queue or release path runs it.
Agent: la-0.71.1a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit f0156ba)
(cherry picked from commit 875a16e)
(cherry picked from commit a6d0260)
…check (PMAT-4971) defect. perf_gate::v3_shape (a50cc1d) is that check as a library; this is its front door: apr test llm shape-check <witness.json> It prints the verdict and every reason, and exits by apr's convention: ADMISSIBLE 0; NOT ADMISSIBLE 5 (ValidationFailed); a missing path or a directory 3; an empty, malformed or bandless witness 4, never 5, so the draft contract's F1 holds across the mapping. It reads a file and needs no runtime, so it routes before the tokio runtime and works in a build without `inference`; the two bench modes take the command by let-else. Tests (commands::test_llm_shape): the control witness exits 0; #4971's falsifier rows (a band at m_formed = c - 1, a divergence before declared_min with no margin) exit 5 naming the band; the same divergence at margin 0.01 exits 0; five unreadable shapes exit 4; missing and directory paths exit 3. Falsifier FALSIFY-V3-SHAPE-CLI-4971. Registry prose for `test` names the new route. validate.rs lists both llm subcommands explicitly (neither names a model file). Not a gate: no merge, queue or release job runs it. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 98cafc2) (cherry picked from commit 6e40a4d) (cherry picked from commit 7888653)
…APR format" (PMAT-4971)
`apr test llm shape-check` mapped an unreadable or empty witness to
`CliError::InvalidFormat`, which prints "Invalid APR format: ..." over a
JSON witness. It is now `InvalidInput` ("Invalid input: V3 shape witness
..."), still exit 4, so RED and NOT ADMISSIBLE (5) never share a code.
The F1 test asserts the variant, the prefix, and that no "APR" appears.
Refs #4971
Agent: la-0.71.1a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit e70987c)
(cherry picked from commit 0c183a9)
(cherry picked from commit e3c85fa)
…ogprob entries (PMAT-4971) V3's S5 admits an early divergence only on a recorded near-tie margin, and until now nothing in Rust could compute one: the margin was a field a witness file carried in. This is the producer's core. - perf_gate::margin: TokenLogprob / TopLogprob read the OpenAI entry (realizar, OpenAI with bytes null, a null logprob, llama.cpp's `id`). token_ids interns entries by text and bytes across all streams, so BatchInvarianceWitness::compare_logprobs hands compare_batch the same ids it would get from token ids. The margin is the gap between the two largest finite logprobs of the m=1 entry at divergence_at, which is the logit gap at temperature 0 (shared log-sum-exp). It is never read from a batch slot and never 0.0 for a step with no runner-up. - llm client: ChatRequest gains `logprobs` / `top_logprobs`, sent only when set (wire_request carries them by struct update). StreamedChatResponse collects every `logprobs.content[]` entry in order, including an entry held to the terminal chunk. The chunk fold moves out of chat_completion_stream into StreamTally (cognitive complexity 69 -> 22), and the loopback fixture's per-script fields into SseScript methods (26 -> 24). Loopback tests cover the request on the socket, the order, null chunks and the margins, plus a MUST-NOT-FIRE where an unset request sends neither key. - Contract: FALSIFY-WITNESS-LOGPROBS-MARGIN-4971, and the shape-check CLI row now names InvalidInput for exit 4. Not a gate. The perf041 port that fires the bands with these fields and writes witness.json is the next slice. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 271d700) (cherry picked from commit a4a064d) (cherry picked from commit 1c96226)
V3-d, first slice: scripts/perf041_batched_parity_probe.py's verdicts, ported to perf_gate::parity with no HTTP. PP-28 refuses a slot whose completion_tokens or logprob entries differ from n_predict; a c>1 band with no m>1 batch in its own log window is UNMEASURABLE; the reference is stable only when two m=1 decodes agree to declared_min; the run passes only with a stable reference, every band decided, and a c>1 band run. A band whose streams never part records divergence_at as the shorter stream's length, not null, so a clean band is admissible to the V3 shape check; a test feeds the producer's witness to check_v3_shape to keep the two in step. The top-2 margin comes from compare_logprobs. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 673f142) (cherry picked from commit d399491) (cherry picked from commit c66f6ab)
V3-d, second slice, under the llm feature: ParityProbe fires the two m=1 references one after the other, then each band's c requests at once, every one asking for logprobs with the sampler's top_logprobs, and reads the server log from the band's own offset for the batch it formed. An unstable reference ends the run with no bands. The verdicts are perf_gate::parity's. Tests replay canned SSE from a loopback server that appends the scheduler's Batch m=N line from inside its handler, as the script's _replay_server does: a clean ladder passes, a box that never batches is UNMEASURABLE, frozen batched slots FAIL, a near-tie flip passes with its margin recorded, an untrusted reference runs no bands. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 5b3bf75) (cherry picked from commit 97546b3) (cherry picked from commit 3c2f2cd)
…(PMAT-4971) The Rust port of scripts/perf041_batched_parity_probe.py now has a command. It fires the probe at --url, reads --server-log for the batch each band formed, prints the verdict, the reference and every band, and writes the witness that `apr test llm shape-check` reads. Exits: PASS 0, FAIL 5 (ValidationFailed), UNMEASURABLE 8 (InferenceFailed). FAIL and UNMEASURABLE stay apart, as the script's 1 and 2 do. A server log, binary or model that is not a file exits 3 before any request is fired. perf_gate::parity::ProbeMatrix reads the policy, the sampler and the ladder from the compiled-in perf-matrix.yaml. The script falls back to its own literals when a key is missing. ProbeMatrix refuses instead and names the key. It also refuses an empty ladder, a 0 in the ladder, a max_constant_run of 0, and a declared_min above n_predict. --ladder replaces only the ladder. The witness records the model by file name only, beside its sha256. An unknown host stays null and is never made up. Not a gate: no merge, queue or release job runs it. Falsifier rows: FALSIFY-PERF041-PORT-4971 (library) and FALSIFY-PERF041-CLI-4971. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 445cdee) (cherry picked from commit e2415f3) (cherry picked from commit 891edf3)
…not --json (PMAT-4971) The parity arg `json: Option<OutputPath>` shared its id with apr's global `--json: bool`, so clap downcast one to the other's type and panicked on every real run: gx10 g12 at 61f0fa5 panicked at extended_commands.rs:2042 (missing log) and lib.rs:160 (refused server) before any request. The unit tests parsed `llm` alone, without the global arg, and passed. The flag is now `--witness <FILE>`. A new test parses through the whole `apr` parser with the witness flag, with the global --json before the subcommand, and with it after, and checks that each lands where it belongs. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 605a9f6) (cherry picked from commit 5308538) (cherry picked from commit 83c76da)
… in the PERF041 row (PMAT-4971) The full-parser test added in e1f24a0 overflowed the default 2 MiB test-thread stack in a debug build on gx10 (SIGABRT), as clap's destructuring of the full `Commands` enum does elsewhere; it now runs on a 16 MB thread, as run_tests_top_k_default_3754.rs does. The same commit's FALSIFY-PERF041-CLI-4971 wording put `apr's` inside a single-quoted YAML scalar, so `pv lint contracts/` failed PV-VAL-001 on apr-serve-openai-compat-v1.yaml; the quote is now doubled. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 9bf1ceb) (cherry picked from commit 9f3bb00) (cherry picked from commit 1b9b223)
…n rule (PMAT-4971) Order step 1 of the V3 bar the C314 quorum adopted on 2026-10-10 06:48Z (receipt quorum-la-0.71.1a-v3bar.md): the draft 0.3.0 moves into contracts/ at 1.0.0 with the bar's statement, falsifier, host classes (lambda RTX 4090 sm_89, gx10 GB10), conditions and order. FALSIFY-SSP-001..016 are re-pointed from the Python reference to the Rust case table in perf_gate::v3_shape. SSP-007 states what main's lambda witness gives today: four reasons, since #4888 refreshed it and c=16 now forms m=16. S7 (FALSIFY-CB-004 read from the witness), the one-near-tie cap and the declared_min floor of 64 are spec-before-code: SSP-017..019 are PLANNED until shape-check reads them. The cap's rationale is the author's wording. Not a gate: no merge, queue or release job runs the check. pv validate and pv lint contracts/ pass; README's CONTRACT_COUNT is regenerated. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e8e238a) (cherry picked from commit 79988b4) (cherry picked from commit 5fae9e4)
…loor (PMAT-4971) The probe now records each request's decode rate from its own token arrivals, and only on a stream the server declared live: the batched CUDA path drops the server's timings. A band's decode_tok_s is the median of its slots' rates, and None when any slot is untimed. shape-check reads the V3 bar's three new rules (C314 round 2): - S7 (FALSIFY-CB-004): per-request decode at c=4 is at least 50% of c=1's. A missing rate at either band is unmeasured, which is not admissible. - the near-tie cap: at most one of c=4, 8, 16 is admitted through the near-tie clause. - the declared_min floor of 64. A test checks that it equals the shipped perf matrix's witness.min_agree_tokens. serving-shape-parity-v1: FALSIFY-SSP-017..019 are no longer PLANNED. SSP-007 now predicts five reasons on main's lambda witness. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a2defb8) (cherry picked from commit 70c6257) (cherry picked from commit 7ed84b6)
…PMAT-4971) At 72283ef on gx10 (aarch64, CPU only), FALSIFY-SSP-007 by hand gave exit 5 with its five reasons, S7 unmeasured among them. The perf_gate, llm::parity and apr-cli test_llm tests all passed. No GPU host class has run the probe yet. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 83ce5b2) (cherry picked from commit d3d31b8) (cherry picked from commit d34127c)
…971) The first mutation run of the Rust V3 shape check (gx10 g16 at 8978bec, perf_gate tests) left three survivors in parity.rs: - `arrivals.len() < 2` in live_decode_tok_s mutated to `==` and `<=`. The clause was redundant: `?` on first/last and `span > 0.0` already give None below two arrivals. It goes, and a two-arrival row pins that two arrivals are enough. - check_reference's `tokens` field could be deleted unnoticed. The reference test now asserts it, including when ref#1 is refused. Not yet built; a gx10 run verifies it before this is pushed. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 6d913ab) (cherry picked from commit 281eed1) (cherry picked from commit 788dcaf)
g17 on gx10 left one survivor: `replace < with <= in check_reference` (parity.rs:186). No test had the two m=1 references agree to exactly declared_min, so the boundary was free. They now diverge at token 4 with declared_min 4, which must read as stable. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 7b42524) (cherry picked from commit 792b150) (cherry picked from commit 4b676a2)
…ns (PMAT-4971) mutation_record said the Rust port had never been mutation-tested. It now carries the gx10 runs: g17 at 5227cda, 98 mutants, 95 caught, 1 missed, 2 unviable; g18 at ba959ef killed that one (5/5 on check_reference). The draft's Python record stays as python_draft. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit d16d43d) (cherry picked from commit 028efff) (cherry picked from commit 8d7477f)
…tokens (PMAT-4971) The quantized fixture model's greedy first token is a stop token, so every reply in this module was empty: three per-token tests failed on completion_tokens == 0, at 4dbdb13 before the recut as well (gx10 g22, 11 passed / 3 failed). With ignore_eos the module runs 14/14 on gx10 (g23, d16d43d plus this change). Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 05cdab9) (cherry picked from commit 1f275af) (cherry picked from commit 4c37a8e)
…pe-check and parity (PMAT-4971) The re-cut onto main brought #4954's TtftVerdict into LlmSubcommand, and V3's explicit match in validate.rs (no wildcard, so a new subcommand that names a model must be handled) did not list it: apr-cli failed to compile with E0004. ttft-verdict reads receipt directories, never a model, so it joins the arm that returns no paths. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 3ab7ebe) (cherry picked from commit f811234)
…ut (PMAT-4971) Regenerated by pv-sat on intel at 1f275af (the m2 re-cut onto main 72282b2); contracts lint_passes_on_real_contracts green on that tree. Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit e38601b) (cherry picked from commit 7cba852)
noahgift
commented
Oct 12, 2026
noahgift
left a comment
noahgift
left a comment
Contributor
Author
There was a problem hiding this comment.
seat 2, aprender-wprodmodels CHANGES @ e102b99 (not an APPROVE)
Reviewed the diff against main 0d607e0: reading only. I did not build; CI owns the build. One sonnet review lane was used, and I checked its findings myself in the head tree.
Blocking
crates/apr-cli/src/commands/serve/handler_gpu_completion.rs:251:handle_gpu_chat_completionhas no logprobs refusal. It is the POST/v1/chat/completionshandler forapr serveon an APR model with CUDA (handlers_include_01.rs:414-423, and the fallbacks at :442/:469). Neither file mentionslogprob. A request{"messages":[...],"logprobs":true,"top_logprobs":3}therefore gets a 200 with no logprobs. That is what ASOC-INV-021 forbids, and it contradicts the PR's "every other route answers 501 and names itself". The fix is the samelogprobs_refusal_response(&req, "APR GPU")that the wgpu, APR CPU and SafeTensors CPU handlers already call, plus a test row.
Should fix or restate
2. crates/aprender-serve/src/gguf/cuda/generate_batched_streaming.rs:602: if use_graph && !recording. With BATCHED_GRAPH=1, a step that records logprobs runs eager instead of the graphed path. Asking therefore changes the kernel path, which session.rs itself says can pick another token on a near-tie. The code comment at :550-552 documents this, and it is inert with the default (graph off). The PR's "asking never changes the token" should either be scoped to graph-off, or the logits should be read after the graphed step.
Nits
- An invalid
top_logprobs(21, -1, 2.5, "3") is a serde 422, not a 400. That is consistent, but the PR text should say so. - The iteration scheduler's
close_done_slotsleaves a done slot's recorder alive until the slot is recycled. Harmless withtry_recv. - The stop-sequence trim may leave logprob entries for tokens cut from the text. Not traced to the end; worth a test row.
Checked and clean: log-softmax subtracts the max and skips NaN; top-k breaks ties by the lower id (matches argmax); a greedy step keeps the device argmax; the non-logprobs scheduler paths looked behaviour-preserving; no workflow, release-script, Makefile or .py change; no new gate wired.
…ing them (PMAT-4971) handle_gpu_chat_completion generated through run_gpu_generation, which computes no logprobs, and answered 200 without them. It now calls logprobs_refusal_response before generating, as the APR CPU, WGPU and SafeTensors routes do. The handler is cuda-only, so the new test reads its shipping source and checks the refusal comes before generation. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-4971) batched_step_token_ids sent a recording step down the eager path whenever BATCHED_GRAPH=1, so one batch mixed graphed and eager steps. The graph is captured against workspace.logits_buf (init_batched_graph_buffers sizes it exactly), and the replay synchronizes before its argmax, so the logits a recording step needs are already there. Both paths now pick their tokens, and a recording step downloads the logits afterwards. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noahgift
commented
Oct 12, 2026
noahgift
left a comment
noahgift
left a comment
Contributor
Author
There was a problem hiding this comment.
seat 1, aprender-df HOLD @ e102b99
The code looks right, and I measured it at this head. The HOLD is only for internal references in the public files. That is the same class that held #5161. No code change is needed.
Privacy: blocking, text-only
New on main (0 files on main carry these today):
- contracts/serving-shape-parity-v1.yaml:30 cites an internal file path outside the repo as the quorum receipt. Drop the path and keep the date and outcome.
- contracts/serving-shape-parity-v1.yaml:14, 60, 61 and 63, and crates/aprender-test-lib/src/perf_gate/v3_shape.rs:681 (the "/nonexistent/..." test path), name an internal agent session. Use a role ("the author", "a non-author session") or a neutral path.
- Internal ruling IDs (C3xx) in public text: contracts/serving-shape-parity-v1.yaml:5, 6 and 35; contracts/apr-serve-openai-compat-v1.yaml:717 and 720; crates/apr-cli/src/commands/test_llm_parity.rs:3; crates/aprender-test-lib/src/perf_gate/parity.rs:7; crates/aprender-test-lib/src/perf_gate/v3_shape.rs:22. Main already has a few of these, so this is lower priority, but these are new.
- The commit messages are squashed by the merge queue (it uses the commit messages, not the PR body). They name build hosts, which the repo's own text also does, so this is advisory only.
Measured at this head (aarch64 host with a GB10 GPU, CPU-pinned lanes)
- aprender-serve --lib logprobs: 40 passed, 0 failed.
- aprender-serve --lib session: 78 passed, 0 failed.
- aprender-serve --lib api::tests: 1685 passed, 0 failed, 3 ignored.
- aprender-test-lib --features llm --lib: 6847 passed, 0 failed.
- apr-cli --lib test_llm: 93 passed, 0 failed.
- aprender-serve --features cuda --lib logprobs_cuda_4971: 1 passed on a real device, with no SKIP line in the log. This covers the direct path, the scheduler m=1 path, the m=2 batch and the iteration scheduler. It matters because the PR body's evidence lists no --features cuda run, and cuda_batch_scheduler, iteration_scheduler, gguf/cuda and the executor modules are all cfg(feature = "cuda"), so none of the non-cuda runs listed compile them.
- Not measured: aprender-orchestrate --lib banco matched 0 tests under that filter. The banco handlers tests ran in neither my run nor, as far as the body says, the author's.
Review notes (advisory, non-blocking)
- first_token_logits.rs reproduces steps 1-4 of prefill_extract_first_token (prefill.rs:1503) instead of sharing them. The "asking never changes the token" invariant then rests on two copies staying identical. Having prefill_extract_first_token call prefill_last_row_logits would make the two the same code.
- In that copy (and in the original), when residual_add_into fails, bias_buf is dropped before std::mem::forget. On that error path, the non-owning view's Drop runs over the borrowed bias allocation. This predates the PR and is carried over.
- generate_batched_streaming.rs:602: under BATCHED_GRAPH=1, a recording step leaves the graphed path for the eager one, and does so for every slot in the batch. The body says a greedy step "never moves to another kernel path". That holds at the default (graph off), but not under the opt-in env. Worth one sentence in the body or contract.
- logprobs.rs top_k_logprobs: select_nth_unstable_by with the same comparator, then sort, is correct. Ties go to the lower id, as ops::argmax does, and the case table checks it.
…ames (PMAT-4971) The contract and the V3 module docs cited an internal review receipt path, working-session names, branch names and internal ruling IDs. They now say what was decided (adopted in review on 2026-10-10, bound to 61f0fa5; the build is Rust only) without them. The CUDA logprobs test finds tinyllama under $HOME/.apr/models instead of two host-specific paths. Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
…MAT-4971) openai_handlers.rs gained the logprobs stream path, so the POST /v1/chat/completions (stream:true) row now cites 919-989,1058-1240, and its stream:true branches at 1336 and 1435 (each line checked against the source it named before). Refs #4971 Agent: la-0.71.1a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
noahgift
commented
Oct 12, 2026
noahgift
left a comment
noahgift
left a comment
Contributor
Author
There was a problem hiding this comment.
seat 2, aprender-wprodmodels APPROVE @ ed43b1e
This re-seats my seat 2 after the CHANGES at e102b99. I read the 4 commits since that head (11 files, +88/-52).
- The blocker is fixed (fd6f25f).
handle_gpu_chat_completionnow callslogprobs_refusal_response(&req, "APR CUDA (AprV2ModelCuda)")after the model check and beforerun_gpu_generation. A new source-reading test asserts the refusal is present and comes before generation. It strips comments and the#[cfg(test)]part first, so it cannot pass on a comment. - The should-fix is fixed (23e42db). With
BATCHED_GRAPH=1, a recording step now stays on the graphed path. I checked the code for the reason given.init_batched_graph_buffersallocatesworkspace.logits_bufbefore capture, and the capture is recorded against it. A failed capture falls back to the eager path, which writes the same buffer.download_logits_atreads that buffer afterbatched_gpu_argmaxsyncs. So the logprobs come from the logits the token was argmaxed from, on either path. - Other changes, all clean:
- The CUDA logprobs test reads the model path from
$HOMEinstead of host-absolute paths. - The serving-shape contract and the parity/v3_shape strings drop internal paths and names.
- The SSE row in
surface_audit.csvis re-derived.
- The CUDA logprobs test reads the model path from
The nits from my first review still stand and do not block: a 422 rather than a 400 for an invalid top_logprobs, and logprob entries that a stop-sequence trim may leave behind. I did not build this head; CI owns the build.
noahgift
commented
Oct 12, 2026
noahgift
left a comment
noahgift
left a comment
Contributor
Author
There was a problem hiding this comment.
seat 1, aprender-df APPROVE @ ed43b1e
Re-seat after my earlier seat on e102b99. I read the four new commits (fd6f25f, 23e42db, fa0af16, ed43b1e): 11 files, +88/-52.
My blocking items are fixed. I scanned the added lines for the internal file path, the session names and the internal ruling IDs I cited before: 0 hits on this head. The host-class names that remain (GPU host labels, evidence/perf041/...) follow what main already carries widely in crates/ and contracts/.
My BATCHED_GRAPH advisory is resolved. I traced batched_step_token_ids on this head:
- A recording step now stays on the graphed path, and the logprobs come from the same logits as the token.
- The graph replay launches, syncs the stream, then
batched_argmax_from_logitsreadsworkspace.logits_buf. download_batched_logitsreadsworkspace.logits_buftoo.- The capture-failure fallback calls
forward_batched_to_token_ids, which also writesworkspace.logits_buf. - On all three paths, the logits downloaded are the ones the argmax chose from. The new doc comments say exactly that.
APR CUDA chat route. handle_gpu_chat_completion now refuses logprobs before run_gpu_generation, instead of answering 200 without them.
- It uses the shared
logprobs_refusal_response, which is in scope because both handler files areinclude!d intohandlers.rs. - The pinning test reads the handler's shipping source, with comments and
#[cfg(test)]cut, and checks that the refusal comes before generation. It reads source text, but it fails on both a missing refusal and a reversed order.
Tests. I did not measure them on this head; no test lane was free. On e102b99 I measured 40/78/1685/6847/93 passing across the logprobs, session, api, test-lib and test_llm filters, plus the CUDA logprobs test on device. The delta since then is the code read above. CI on this head is the test evidence for it, and I did not build the cuda-only change myself.
Still open from my first seat, non-blocking: first_token_logits keeps its own copy of the prefill steps instead of sharing them.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
V3 serving-shape parity for #4971, train 0.71.1: the server's logprobs, and the probe and shape check that read them. This PR adds the measurement only and wires no gate: V3 as a gate needs its own ruling.
/v1/chat/completionsreadslogprobsandtop_logprobs(0..=20). Before this, serde dropped both keys, so a client that asked for them got a 200 without them. Each entry carries the raw logit, which the V3 top-2 margin needs.BATCHED_GRAPH=1: a recording step stays on the graphed path and reads the logits buffer the graph was captured against.top_logprobs(21, -1, 2.5, "3") is a 422 from deserialization;top_logprobswithoutlogprobs: trueis refused with a 4xx naming the field.apr test llm parityruns the perf041 probe overLlmClientand writes a PP-26 witness.apr test llm shape-checkreads a witness against the V3 bar incontracts/serving-shape-parity-v1.yaml: S1-S7, eps 0.05, the near-tie cap of 1 and the declared_min floor of 64. A RED witness exits 4.Recut onto main
This is
la-711a/4971-v3a-marginrecut onto main 0d607e0 as the new branchla-711a/4971-v3a-margin-m4, never merged with main. Head ed43b1e (33 commits; the last three answer review, see below). It is the-m3recut (onto cc71bde) replayed onto main 0d607e0; the only file main and this branch both change since cc71bde isdocs/roadmaps/roadmap.yaml, regenerated bymake roadmap-aggregate. Hand resolutions against main, made on the-m3recut:Session::generate_innertakestop_n, and the newSession::generate_traced_with_logprobstakes both.generate_tracedandgenerate_with_logprobsare thin wrappers over it, so there is one timing path.build_chat_response_tracedtakeslogprobslast. The non-streaming Qwen3.5 turn lives inqwen35_turnand carries main's trace and layer timings with the logprobs.PhaseTimingsreceiver besideStreamLogprobs.LlmSubcommandholdsTtftVerdict,ShapeCheck,ParityandBench, andvalidate.rs's explicit model-path match lists all four (3ab7ebe; the first build failed E0004 without it).session.rsmerged clean around main'sadmit_and_reserve/prepare_turn;session_tests.rskeeps main'sprepare_turntest and this PR'slogprobs_4971module side by side.aprender-test-lib's client,StreamAccumcollects the streamed logprob entries (main had replacedStreamTallywith it).The ONT witness is re-stamped by pv-sat on the recut, as the last commit; pv-sat finds it fresh at e102b99 and again at fa0af16 (the review fixes change no contract the witness covers).
Review fixes (fd6f25f, 23e42db, fa0af16, ed43b1e)
handle_gpu_chat_completionanswered 200 without them, becauserun_gpu_generationcomputes none. It now returnslogprobs_refusal_response(&req, "APR CUDA (AprV2ModelCuda)")before any generation. The testthe_cuda_chat_route_refuses_logprobs_before_it_generatesreads the shipping handler and requires the refusal beforerun_gpu_generation. Deleting the refusal turns it RED (mutant run below).BATCHED_GRAPH=1, a recording step ran eager, so asking could change the kernel path. It now takes the graphed path and downloads the logits after the replay. The graph is captured againstworkspace.logits_buf, whichinit_batched_graph_bufferssizes to exactly m*vocab, sodownload_batched_logitsreads the replay's logits. Not measured on a device withBATCHED_GRAPH=1; the default is graph off.$HOME/.apr/models.docs/audits/surface_audit.csv: the chat SSE row cites the moved lines inopenai_handlers.rs(each re-checked against the source line it named before);check_dogfood_coverage.shG2.1 passes.Evidence on x86_64 at fa0af16, checked out by sha:
apr-cli --lib commands::serve::336 passed, 0 failed. Mutant (the refusal deleted): the new test FAILED, as it must.aprender-serve --lib logprobs40 passed, 0 failed.cargo check -p apr-cli -p aprender-serve --features apr-cli/cuda,aprender-serve/cudarc 0; clippy-D warningsrc 0 with and without cuda.aprender-test-lib --features llm(perf_gate, llm) 543 passed. pv-sat: no witness change.Reviewer's run on a CUDA device (at e102b99):
aprender-serve --features cuda --lib logprobs_cuda_49711 passed with no SKIP line, covering the direct path, the scheduler's m=1 path, the m=2 batch and the iteration scheduler. Not measured: the banco handler tests inaprender-orchestrate(the filter matched 0 tests).CI at e102b99:
falsify_chromium_driver_is_realfailed 3/3 on a runner with no browser ("Browser not found"); this PR does not touch the browser driver.Evidence
perf_gate/v3_shape.rspluslive_decode_tok_s,band_decode_tok_sandcheck_reference: 98 mutants, 0 survivors after the follow-up tests (recorded in the contract'smutation_record).aprender-serve --lib16286 passed, 0 failed, 64 ignored, 4 filtered out (the four long tests, not_measured here).aprender-test-lib --features llm543 passed. pv-sat regenerated the witness (committed e38601b), andaprender-contractslint_passes_on_real_contracts passes.apr-cli --lib(test_llm, extended_commands, cli_commands, validate) 303 passed, 0 failed. Clippy-D warningsis clean on aprender-serve, apr-cli and aprender-test-lib (llm).apr-cli --test cli_commands15 passed.aprender-serve --lib16316 passed, 0 failed, 64 ignored, the same 4 filtered out.apr-cli --lib303 passed. Clippy clean.aprender-test-lib --features llm543 passed. pv-sat: witness fresh. Contracts lint passes.apr-cli --test cli_commands15 passed.-m3head, on main cc71bde): the same steps, the same counts.--no-default-featuresfailed to build incommands/chat.rsandcommands/serve/route_index.rs, files this PR does not touch. Main fails the same way (b350e83: 23 errors at the same four sites); that is LAB: apr-cli builds without inference (cargo check -p apr-cli --no-default-features --lib) #4972, now fixed on main.ignore_eos, because the fixture model's greedy first token is a stop token. Without it every reply was empty, and the per-token checks saw nothing. With it, the logprobs tests pass 14/14.ont-delta: type serving-shape-parity-v1
keep-open: #4971 stays open after this PR. Its acceptance is the perf041 witness on the reference GPU host for Qwen3.5-4B at c in {1,4,8,16}, and S3 cannot pass while apr serve's Qwen3.5 path is single-stream (#5046), which this PR does not change.
Refs #4971, #5046
🤖 Generated with Claude Code