Repository navigation
R-6: agent conformance suite: runner, OpenAI + Anthropic cases, receipt per case × model × backend #5349
Description
Refs #4000 (EPIC 0.72 "Agent Ready", row R-6). Every other row's done_when runs through this suite, so it lands first.
Defect. No conformance suite exists. contracts/apr-serve-openai-compat-v1.yaml describes the OpenAI surface as obligations, but nothing runs a fixed set of client-observable cases against a live apr serve per model × backend and reports a cell per case. There is no Anthropic shape at all.
Done when:
- Runner. One runner sends each suite case to a running
apr serve(any model, any backend) and writes one receipt cell per case × model × backend:pass,fail(with the failing assertion), ornot_measured(with the reason). A cell that is absent is a missing measurement, and the run exits non-zero. - Cases. The OpenAI cases come from the existing contract: non-streaming and SSE chat,
finish_reason(stopandlength),usage, completions, models. The Anthropic cases live in a newcontracts/apr-serve-anthropic-messages-v1.yaml. Until R-1 adds/v1/messages, they are recorded asfail(404), never skipped. At least one case per row R-1..R-5 per shape. - Prove. A planted mutant that drops
finish_reasonturns the run RED. The same run without the mutant is green on the cases the server supports. - Nightly. CPU, qwen3-1.7b-q4km, on an idle host, with the GPU certified matrix nightly. It runs as LAB until it is promoted.
- PR job. A CI job on PRs that touch
aprender-serveis a new gate. It needs its own ruling, three green nights and a retired gate, and is not part of this issue's first PR.
Activity
noahgift commentedon Oct 14, 2026
S1 landed on the branch
feat/r6-conformance-5349@601f9ae866(no PR yet; waits for a slot after the 0.72 cut).realizar::api::conformance: 12 cases (7 OpenAI, 5 Anthropic) covering R-1..R-5 on both shapes, one cell per case × model × backend (pass/fail/not_measured, no skip; absent cells named bymissing_cells).serve_conformanceexample: runs the suite over HTTP against a liveapr serveand writes the receipt JSON. It exits 1 on any fail, not_measured or missing cell.contracts/apr-serve-anthropic-messages-v1.yaml, FALSIFY-ASM-001..007: 7/7 pass; clippy-D warningsclean on lib, tests and the example.finish_reasonstripped from JSON and SSE):oai.chat.finish_reasonandoai.chat.streamgo RED.Baseline on the demo router: 5/12 pass.
tool_calls/v1/messagesrouteNext: S2, the nightly runner on qwen3-1.7b-q4km CPU plus the GPU certified matrix; first receipt 10-16.
fix(serve): serve_conformance usage names qwen3.5, not a stale family (…
feat(serve): R-6 S2 nightly runner — apr serve + serve_conformance re…
test(serve): R-6 conformance case for a streamed tool call (Refs #5349,
noahgift commentedon Oct 15, 2026
R-6 S2: first live receipt from
scripts/serve_conformance_nightly.sh(branchfeat/r6-conformance-5349@ f3c5a31).apr 0.71.3 (f3c5a312d9),apr serve runon CPU (--no-gpu), Qwen3.5-0.8B-Q4_K_M (sha256bd258782e35f…dc517), 2026-10-14 21:06–21:10Z. Server up; 17 cells, 0 missing, 0 not_measured.api::conformance8/8, contractsstale_examples+lint_passes_on_real_contracts2/2.8/17 pass
tool_calls[0].function.name""/v1/messagesroute (R-1)Caveat on R-3: one prompt is one input. A greeting prompt may be near-deterministic even under real sampling, so before the seed FAIL is read as "sampling is greedy" the case needs a higher-entropy prompt or a second prompt. The check still fails honestly here; the cause is not yet proven.
Not a gate (LAB). S3 (a CI job) goes to a quorum after S2, not before 10-19.
noahgift commentedon Oct 22, 2026
Tracked in #5348 (triage 2026-10-22).
chore(serve): carry PMAT-Ticket trailer for the R-6 conformance suite…
ci(serve): wire the conformance runner's --self-test into ci/sections…
noahgift commentedon Oct 25, 2026
R4 triage parked and closed PR #6118 (feat(serve): agent conformance suite runner + Anthropic Messages contract (R-6, Refs #5349)). This ticket stays open and links it; re-cut or reopen the PR when rebased.