PMAT-3739: CRUX inference dogfood slice 1 — the same GGUF + golden prompts through apr, llama.cpp@pin and ollama on run; a competitor right and apr wrong is RED

docs/roadmaps/entries/PMAT-3739.yaml

@@ -0,0 +1,24 @@
+- id: PMAT-3739
+  github_issue: 3739
+  item_type: task
+  title: 'CRUX inference dogfood slice 1: the SAME GGUF + SAME golden prompts through apr, llama.cpp@pin and ollama on the `run` verb, judged by one rule: a competitor right and apr wrong is RED'
+  status: in_progress
+  priority: critical
+  assigned_to: aprender-76
+  created: 2026-09-21T17:10:00Z
+  updated: 2026-09-21T20:20:00Z
+  spec: null
+  acceptance_criteria:
+  - 'SLICE 1 of #3739, as the design note on the issue lays out (issuecomment-5764437831): the `run` verb, the golden prompts, thinking OFF, lambda. This PR does NOT meet the issue''s done_when in full, and it says `Refs #3739`, never a closing keyword. Every receipt it writes carries a `not_covered` list naming the verbs chat, serve and code, thinking ON, the consumer-brief rungs and max context, and TTFT, so a partial run cannot be read as a full one.'
+  - 'done_when 1, the part this slice meets, verbatim: "`scripts/crux_inference_dogfood.sh <version>` runs ... the SAME prompt set (the golden prompts ...) through: run: `apr run` · `llama-cli` · `ollama run`". Measured: `bash scripts/crux_inference_dogfood.sh 0.69.0 --backend cpu --model ~/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf` on lambda gives rc 0 with 3 cells GREEN, apr and llama.cpp@d1d3c3396 both correct. It declines (rc 2) when `apr --version` is not <version>, when no --model is given, for a verb other than run, when --engines omits apr, and when no comparator resolves.'
+  - '"Same model" is recorded, not asserted, verbatim: "ollama IMPORTS the exact GGUF via a Modelfile (`FROM <path>`) and the receipt records sha256 for all three engines". The receipt holds the input sha256 and ollama''s stored blob digest, with `blob_is_input`. An ollama import that carries no chat template (`TEMPLATE {{ .Prompt }}`, what 0.5.7 does on lambda) is REFUSED by name for every cell, never run as a raw completion against two engines that chat. GPU sharing, per the cop''s ruling: on the gpu lane one cell (one prompt through every engine) holds /tmp/apr-gpu.lock (fd 9, closed in every engine) for its engine invocations only. ollama runs with `--keepalive 0`, and the cell stops the model and waits for `ollama ps` to drop it before releasing the lock, recording `vram_released`. The cpu lane takes no lock (the rule''s clause 2); every engine runs under `choom -n 1000`.'
+  - 'done_when 2, the part this slice meets: per cell the receipt records answer correct (golden), exact-match agreement across engines, token-id parity (apr''s `-v` prompt ids against llama.cpp''s own `/apply-template` + `/tokenize` rendering of the same messages; count plus the id prefix apr prints, labelled `apr_ids_complete`), and prompt/completion tokens and rates as each engine REPORTED them, labelled by engine and never judged. The driver reads no clock and computes no rate (`check_no_competing_harnesses.sh` stays count=0). TTFT and max context belong to later slices and are listed as not covered.'
+  - 'done_when 3, the gate, verbatim: "a cell where llama.cpp OR ollama answers correctly and apr doesn''t is **RED** (a finding naming the cell)". `bash scripts/check_crux_inference_judge.sh`, which guard_tree runs bare, gives rc 0 and "21 ok, 0 broke". Its row 2 is the #3715 falsifier: llama.cpp right and apr wrong makes the judge exit 1 with the cell RED. Other rows make RED an apr that exits non-zero, falls back off the lane''s backend, or is MISSING. ANY UNJUDGED cell (no comparator answered) declines the run with rc 2, because under the 17:19Z amendment "a missing cell is a NO-GO". ALL_WRONG (every engine wrong) is NAMED and counted per model (`summary.all_wrong_by_model`), never a violation and never a pass, per the cop''s ruling. The prompt set must declare a POSITIVE CONTROL (golden-2plus2, `"control": true`). A control cell that is ALL_WRONG, a model with no measured control cell, and a set that declares none each decline rc 2, with the cause named in `declined_because`. A comparator that exits non-zero or cannot be parsed is no oracle. A drift row pins the prompt set to golden_output.rs::golden_test_cases.'
+  - 'Mutation, 11 mutants, each turning the table RED: removing the RED rule breaks 6 rows; ignoring apr''s exit code breaks 2; ignoring fallback/backend breaks 1; skipping a missing apr row breaks 14; letting UNJUDGED pass breaks 4; `correct()` ignoring `answered` breaks 4; letting a control ALL_WRONG pass breaks 2; letting an undeclared control pass breaks 1; letting an unmeasured control pass breaks 1; not counting ALL_WRONG per model breaks 1; adding a pattern to the prompt set breaks 4, including the drift row. (An earlier no-GREEN decline was removed, not mutated: the per-model control rule subsumes it, and a mutant of it broke nothing.)'
+  - 'The first receipts are committed at evidence/crux/0.69.0/lambda-cpu.{json,md} and evidence/crux/0.69.0/lambda-gpu.{json,md}, measured by apr 0.69.0 and the harness at afb05a3de (`apr --version` and `harness.sha` in each receipt). MEASURED: lambda-cpu is PASS, 3 GREEN, with apr and llama.cpp both correct. lambda-gpu is RED, 2 RED + 1 GREEN: on golden-2plus2 (the control) and golden-paris, `apr run --gpu` exits 14 (CUDA diverges from CPU, cosine 0.8795 at position 3; wgpu rejected at 0.955; fell back to CPU and refused) while llama.cpp answers correctly on the same RTX 4090. golden-greeting ran on the GPU with no fallback and is GREEN. The RED cells are an instance of the open #3602 (same model, same host: the GPU result fails its own correctness check and falls back). They are named on #3739 and are not fixed in this PR. The harness''s job is to go RED here, and it did.'
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - kind:code
+  notes: 'Refs #3739, slice 1 (not a closing keyword; the issue has 6 done_when items and this meets parts of 1, 2 and 3). SCOPE AMENDMENT on the issue (issuecomment-5764587628, cop aprender-04, 17:19Z), verbatim: "Operator, verbatim: \"and CRUX for each verb in pre-release dogfood\". This applies the earlier operator line, \"the way to prevent testing leakage is tests are dervived from surface and SHACL enforced\" (#3745). The verb universe is DERIVED, not listed. It is every command in the surface the release-candidate binary emits (#3745 S1). The run/chat/serve/code list in this issue''s title is superseded as a scope; those four are the first slice, not the set. Every derived verb has a CRUX mapping entry in one committed correspondence file ... A derived verb with NO entry is a SHACL violation. Absence is never \"no comparator\". ... Pre-release, and it gates the tag. CRUX runs in the pre-publish dogfood (T-1), and its receipt is a `:CruxCell` input to `release-readiness-v1` (#3715/#3745). A cell where a comparator is right and apr is wrong is a NO-GO, and so is a missing cell." The cop''s ruling on sequencing, relayed the same minute: "The thin slice (qwen2.5-coder-1.5b `run`, lambda, 3 engines) stays first", and the correspondence file and `:CruxCell` fields are agreed with aprender-97 (#3745 S2, #3715) BEFORE they are coded. So this PR adds no correspondence file. The cop''s rulings of 18:47Z, verbatim: "(1) ALL_WRONG is NAMED, not a violation, since the bar is \"apr right wherever a competitor is right\". Two guards so it can''t become the escape: (i) it''s never scored Pass, and the receipt counts ALL_WRONG per model; (ii) every model''s prompt set includes a positive-control prompt that the smallest model answers right on every engine. If that control comes back ALL_WRONG, the run is DECLINED (rc 2) as a broken harness, never recorded as a model limitation." and "Every CRUX ollama request sets `keep_alive: 0`, runs under the GPU lock, and the cell checks VRAM is released before it drops the lock." Its ruling (2), the lambda ollama upgrade as a binary-only forjar change in paiml/infra, is slice 2 and is not in this PR. Its receipt already keys cells on the release-readiness tuple (model_sha256, host, verb, thinking, rung) plus prompt_id, as aprender-97 asked, and makes an UNJUDGED (unmeasured) cell decline. The issue''s done_when, verbatim: (1) "`scripts/crux_inference_dogfood.sh <version>` runs, per inventory Q4_K model × {lambda, gx10}, the SAME prompt set (the golden prompts + a consumer brief from #3716 at each declared context rung that fits) through: run: `apr run` · `llama-cli` · `ollama run`; chat (multi-turn): `apr chat` · llama-cli conversation · `ollama run` multi-turn; serve: `apr serve` · `llama-server` · `ollama serve`, all through the SAME OpenAI `/v1/chat/completions` client, streaming + non-streaming; code: `apr code` on a scripted edit-and-verify task, and the same task driven through llama-server and ollama by the same client; thinking ON and OFF where the model supports it (#3723), at temperature 0, same seed" (2) "per cell it records: answer correct (golden), exact-match agreement across engines, token-id parity (apr vs `llama-tokenize`), TTFT, decode tok/s, prompt/completion tokens, and the max context each engine accepts before refusing (or failing silently)" (3) "the gate: a cell where llama.cpp OR ollama answers correctly and apr doesn''t is RED (a finding naming the cell). Performance ratios are REPORTED, never claimed (the withdrawn-headline rule). The receipt feeds the SHACL shape (#3715) as a required cell kind." (4) "ollama is upgraded on lambda through the declared path (forjar) to a version that loads qwen35 and qwen3moe GGUFs; both hosts'' versions are recorded" (5) "runs in the release train (the T-1 `models` step, #3717) and ALSO on demand; the results table is committed per release (`evidence/crux/<version>/`)" (6) "depends on: #3563 (A), #3723, #3571". SLICING, ruled by the cop (aprender-04, 2026-09-21: "Start with a design note on #3739 and a first thin slice (qwen2.5-coder-1.5b Q4_K_M, `run` verb, lambda, all 3 engines) to prove the harness, then widen") and laid out in the design note: 1 run (this PR); 2 ollama on lambda via forjar + gx10; 3 chat; 4 serve through one client (perf via `apr test llm bench`, needs #3571); 5 code; 6 thinking ON (needs #3723); 7 consumer rungs + max context; 8 T-1 wiring (#3717). SCOPE: scripts/crux_inference_dogfood.sh (driver), scripts/lib/crux_inference_judge.py (judge), scripts/check_crux_inference_judge.sh (case table), scripts/crux_inference_prompts.json (prompt set), evidence/crux/0.69.0/*, this fragment. ONE DELIBERATE DEVIATION from the letter of done_when 2: token-id parity uses llama.cpp''s server `/tokenize` on its own `/apply-template` rendering, not the `llama-tokenize` binary. lambda''s pinned build dir (the one llama_bin.sh resolves) has no `llama-tokenize`, the server exists on both hosts, and both call the same llama_tokenize on the same vocab. Rendering first also makes parity cover the chat template, which is where #3672 lives. PERF-009: the driver starts engines but contains no rate token and reads no clock, and the judge starts nothing, so the competing-harness count stays 0 without an allowlist entry. The serve slice takes TTFT and decode rate from `apr test llm bench`, not from a new timer. FINDINGS on lambda, apr 0.69.0 @ 52f43da71, qwen2.5-coder-1.5b Q4_K_M sha256 cc324af0…: (a) apr renders the chat template twice (#3672): 46 prompt ids against llama.cpp''s 36, diverging at id 1, and the U+200B escapes encode as id 0 (#3726). The golden answer is still right ("4"), so only the parity field sees it. (b) ollama 0.5.7 imports a GGUF with no chat template; gx10''s 0.33.2 imports the GGUF''s own jinja template but stores blob b9a2f9f3…, not the input''s cc324af0…'