Skip to content

PMAT-3718: apr run --json reports prompt_tokens/completion_tokens/finish_reason; serve never reports a cut as a stop - #3753

Closed
noahgift wants to merge 5 commits into
mainfrom
feat/3718-run-json-usage
Closed

noahgift wants to merge 5 commits into
mainfrom
feat/3718-run-json-usage

Conversation

@noahgift

@noahgift noahgift commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Closes #3718

What changes

apr run --json, including the --stream final event, now reports:

  • prompt_tokens: the count after the chat template is applied, BOS included, i.e. what the model actually read.
  • completion_tokens: the engine's own count, never the word-count stand-in tokens_generated falls back to.
  • finish_reason: "stop" or "length".
  • context_length: from the model's metadata.

Any of these that a path did not report is null, never 0.

How it's computed:

  • New entry point. realizar gains infer::run_inference_report → (InferenceResult, RunReport). It is a separate entry point, so none of the ~75 InferenceResult literals change.
  • Why generation ended is computed where the stop set and the budget are known. Every decode loop was traced. The qwen35, MoE and wgpu loops push the stop token; the dense CPU and CUDA loops break before pushing it. Only the dense CPU loop clamps the budget to the room left in the context.
  • One FinishReason. It is declared once, in infer::run_report; api::types::FinishReason re-exports it.

Serve already returned an OpenAI-shaped usage with the post-template count. Three silent cuts are fixed:

  1. The dense CPU backend judged finish_reason against the requested max_tokens after clamping the run to the context, so a clamped cut said "stop". It is now judged against the budget the run actually had, streaming or not.
  2. A prompt that cannot fit was a 500, or, when streaming, an HTTP 200 carrying an SSE error that still ended "stop". It is now a 400, refused before generation.
  3. The APR CPU fallback hardcoded "stop". When nothing was generated, it also returned the prompt as the completion.

prefill_ms is deliberately not in this row. aprender-3e and I agreed its definition; it arrives with #3606 and 3e's #3596 timer.

Evidence

Real model (lambda, CPU, Qwen3.5-0.8B-Q4_K_M, pinned apr 0.69.0 (3fadee114)):

  • prompt_tokens 24. The same templated string through llama.cpp's llama-tokenize gives the same 24 ids, id for id.
  • A 6-token budget on a poem prompt: 6 tokens, "length".
  • A short answer: "stop".

Case row in CI: a fixed text prompt through the GGUF tokenizer. The expected ids <s> ▁Hello ▁world ! = 4 are derived from the vocabulary, not from apr's encoder.

Serve router tests (api::tests::usage_finish_3718): clamped cut → "length" (both modes); an unfittable prompt → 400 (both modes); stop under budget → "stop".

Mutants, each red: serve clamp reverted; 400 reverted; report clamp off; always-length; never-length; the prompt count off by one.

Suites and checks:

  • aprender-serve lib: 15914 passed
  • apr-cli lib: 7301 passed
  • aprender-contracts lib: 1684 passed
  • clippy -D warnings on both crates and fmt: clean
  • AD-04 quorum: 2 of 3 seats by distinct model id, docs/audits/quorum-PMAT-3718.json.
    Corrected by the cop 2026-09-22: the three lanes were gemini-3.1-pro-high x2 plus
    gemini-3.1-pro-low, and under PMAT-125 two lanes sharing a model id count as ONE seat.
    The artifact's own partial_reasons said so; this body did not. The row is an ancestor
    of release/0.69.1-batch-1 and ships inside that batch's receipt, so the claim is corrected
    here rather than by a new head. Found by aprender-a8 during the 2026-09-22 wind-down.

The full record is in docs/roadmaps/entries/PMAT-3718.yaml. Per the batching default this PR is not armed: it is for the cop to fold.

🤖 Generated with Claude Code

noahgift and others added 5 commits September 21, 2026 18:26
…on; serve never reports a cut as stop

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…F tokenizer, count derived from the vocab

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…calar)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…emini-3.1-pro-low; 2 distinct ids recorded)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3753 head=a814867c955ec19ead85b75a5832b610e9129d61 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

@noahgift

Copy link
Copy Markdown
Member Author

Superseded — this PR's content is already on main (operator 1-day PR rule triage, 2026-09-24).

Carrying commit: 8c7822f — release(0.69.1): batch 1 (#3772).
Evidence (measured against origin/main @aa7c6ef03): 810/810 added lines present on main; 18/24 files byte-identical, the rest edited further on main. #3718 stays open for its own acceptance.

Branch is NOT deleted: feat/3718-run-json-usage @ a814867c9.

@noahgift noahgift closed this Sep 24, 2026
@noahgift
noahgift deleted the feat/3718-run-json-usage branch October 9, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:code Work is a code change (derived rule, #4159)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

apr run --json: add prompt_tokens (and completion_tokens) so consumers can see how many tokens the model actually read — RAH asked (#3716)

1 participant