noahgift · Sep 21, 2026
diff --git a/docs/roadmaps/entries/PMAT-3795.yaml b/docs/roadmaps/entries/PMAT-3795.yaml
@@ -0,0 +1,22 @@
+- id: PMAT-3795
+  github_issue: 3795
+  item_type: task
+  title: 'CRUX slice 3 (#3739): the chat verb — a two-turn conversation through apr chat, llama.cpp and ollama (pty-driven), hf and llamafile; judged on the final turn'
+  status: in_progress
+  priority: critical
+  assigned_to: aprender-76
+  created: 2026-09-21T23:00:00Z
+  updated: 2026-09-21T23:20:00Z
+  spec: null
+  acceptance_criteria:
+  - 'Issue #3795 done_when 1, verbatim: "`scripts/crux_inference_dogfood.sh <version> --verbs run,chat` runs a multi-turn conversation per model through every engine. A `chat` request without `run` declines, because the positive control lives in `run`." The prompt is `chat-arith-2turn`, the user turns "What is 2+2?" then "Now multiply that by 3.", judged on the FINAL reply (`expect_any ["12"]`); every reply is recorded.'
+  - 'done_when 2, verbatim: apr runs through `apr chat`, one user turn per stdin line; llama.cpp and ollama through `scripts/lib/crux_pty_chat.py`, a stdlib pseudo-terminal driver, because they do not take turns from a pipe. MEASURED before building it: llama-cli fed a pipe loops on empty `> ` prompts until killed; `ollama run` reads the whole pipe as ONE prompt and answers both turns in one reply. Driven through the pty helper, each held the conversation: llama-cli and ollama both gave ["2 + 2 equals 4.", "4 * 3 equals 12."]. hf and llamafile use `gen --verb chat` with the user turns (infra-3c, a3eecfe2d).'
+  - 'done_when 3, verbatim: "The judge scores the final turn with the same RED rule. `apr chat` reports no backend (#3794), so apr''s chat cells carry `backend_verified: false` and the receipt''s `not_covered` names it. The run-verb token-parity field isn''t reported as measured for chat."'
+  - 'done_when 4, MEASURED: `bash scripts/check_crux_inference_judge.sh` gives 42 ok, 0 broke. The chat rows: a GREEN chat; apr losing the first turn''s answer ("9") while llama.cpp carries it ("12") is RED; an apr transcript with no `Assistant:` turn is RED; and the record of every turn, `backend_verified: false`, and "not measured for the chat verb". Mutants: judging apr chat on the FIRST turn breaks 1; not recording backend_verified breaks 1; measuring chat as run breaks 1.'
+  - 'done_when 5, MEASURED on lambda (cpu lane, qwen2.5-coder-1.5b Q4_K_M, `--verbs run,chat`): PASS, 4 cells (3 run + 1 chat), 4 GREEN. In the chat cell apr, llama.cpp@d1d3c3396 and ollama 0.33.2 each gave ["2 + 2 equals 4.", "4 * 3 equals 12."]. hf and llamafile were refused by name (their drivers are on PMAT-3778, not yet merged). `not_covered` names apr chat''s unverified backend (#3794). No ollama import was left behind.'
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - kind:code
+  notes: 'Refs #3739 (slice 3; not a closing keyword for the parent). Closes #3795 when this merges. STACKED on crux-3774-verb-correspondence (#3789), so the diff to judge is `origin/crux-3774-verb-correspondence...HEAD`. SCOPE: scripts/crux_inference_dogfood.sh (the chat verb), scripts/lib/crux_pty_chat.py (new), scripts/lib/crux_inference_judge.py (chat parsing), scripts/check_crux_inference_judge.sh (chat rows), scripts/crux_inference_prompts.json (the chat prompt; the golden drift check covers only the golden rung), and this fragment. FOUND while building it: #3794, where `apr chat --no-gpu` still uploads the GGUF to CUDA (VRAM held outside the GPU lock; generation honours force_cpu, the load path does not) and apr chat reports no machine-readable backend. The pty helper starts only the argv it is given and reads no clock for a metric (its per-turn timeout is a bound), so the competing-harness count stays 0.'
diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml
@@ -20021,3 +20021,25 @@ roadmap:
   labels:
   - kind:code
   notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml, evidence/crux/hf-sources.yaml, scripts/lib/crux_inference_judge.py, scripts/crux_inference_dogfood.sh (the plugin-engine hook), scripts/check_crux_inference_judge.sh, and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.'
+- id: PMAT-3795
+  github_issue: 3795
+  item_type: task
+  title: 'CRUX slice 3 (#3739): the chat verb — a two-turn conversation through apr chat, llama.cpp and ollama (pty-driven), hf and llamafile; judged on the final turn'
+  status: in_progress
+  priority: critical
+  assigned_to: aprender-76
+  created: 2026-09-21T23:00:00Z
+  updated: 2026-09-21T23:20:00Z
+  spec: null
+  acceptance_criteria:
+  - 'Issue #3795 done_when 1, verbatim: "`scripts/crux_inference_dogfood.sh <version> --verbs run,chat` runs a multi-turn conversation per model through every engine. A `chat` request without `run` declines, because the positive control lives in `run`." The prompt is `chat-arith-2turn`, the user turns "What is 2+2?" then "Now multiply that by 3.", judged on the FINAL reply (`expect_any ["12"]`); every reply is recorded.'
+  - 'done_when 2, verbatim: apr runs through `apr chat`, one user turn per stdin line; llama.cpp and ollama through `scripts/lib/crux_pty_chat.py`, a stdlib pseudo-terminal driver, because they do not take turns from a pipe. MEASURED before building it: llama-cli fed a pipe loops on empty `> ` prompts until killed; `ollama run` reads the whole pipe as ONE prompt and answers both turns in one reply. Driven through the pty helper, each held the conversation: llama-cli and ollama both gave ["2 + 2 equals 4.", "4 * 3 equals 12."]. hf and llamafile use `gen --verb chat` with the user turns (infra-3c, a3eecfe2d).'
+  - 'done_when 3, verbatim: "The judge scores the final turn with the same RED rule. `apr chat` reports no backend (#3794), so apr''s chat cells carry `backend_verified: false` and the receipt''s `not_covered` names it. The run-verb token-parity field isn''t reported as measured for chat."'
+  - 'done_when 4, MEASURED: `bash scripts/check_crux_inference_judge.sh` gives 42 ok, 0 broke. The chat rows: a GREEN chat; apr losing the first turn''s answer ("9") while llama.cpp carries it ("12") is RED; an apr transcript with no `Assistant:` turn is RED; and the record of every turn, `backend_verified: false`, and "not measured for the chat verb". Mutants: judging apr chat on the FIRST turn breaks 1; not recording backend_verified breaks 1; measuring chat as run breaks 1.'
+  - 'done_when 5, MEASURED on lambda (cpu lane, qwen2.5-coder-1.5b Q4_K_M, `--verbs run,chat`): PASS, 4 cells (3 run + 1 chat), 4 GREEN. In the chat cell apr, llama.cpp@d1d3c3396 and ollama 0.33.2 each gave ["2 + 2 equals 4.", "4 * 3 equals 12."]. hf and llamafile were refused by name (their drivers are on PMAT-3778, not yet merged). `not_covered` names apr chat''s unverified backend (#3794). No ollama import was left behind.'
+  phases: []
+  subtasks: []
+  estimated_effort: null
+  labels:
+  - kind:code
+  notes: 'Refs #3739 (slice 3; not a closing keyword for the parent). Closes #3795 when this merges. STACKED on crux-3774-verb-correspondence (#3789), so the diff to judge is `origin/crux-3774-verb-correspondence...HEAD`. SCOPE: scripts/crux_inference_dogfood.sh (the chat verb), scripts/lib/crux_pty_chat.py (new), scripts/lib/crux_inference_judge.py (chat parsing), scripts/check_crux_inference_judge.sh (chat rows), scripts/crux_inference_prompts.json (the chat prompt; the golden drift check covers only the golden rung), and this fragment. FOUND while building it: #3794, where `apr chat --no-gpu` still uploads the GGUF to CUDA (VRAM held outside the GPU lock; generation honours force_cpu, the load path does not) and apr chat reports no machine-readable backend. The pty helper starts only the argv it is given and reads no clock for a metric (its per-turn timeout is a bound), so the competing-harness count stays 0.'
diff --git a/scripts/check_crux_inference_judge.sh b/scripts/check_crux_inference_judge.sh
@@ -63,12 +63,12 @@ ollama_out() { # ollama_out <dir> <pid> <answer>
 }
 row() { # row <manifest> <engine> <pid> <rc> <stdout> <stderr> [refused]
   python3 - "$@" <<'PY'
-import json, sys
+import json, os, sys
 m, eng, pid, rc, o, e = sys.argv[1:7]
 ref = sys.argv[7] if len(sys.argv) > 7 else ""
 open(m, "a").write(json.dumps({"kind": "gen", "engine": eng, "prompt_id": pid, "rc": int(rc) if rc else None,
     "stdout": o or None, "stderr": e or None, "refused": ref or None, "model_sha256": "%s",
-    "host": "fixture", "verb": "run", "thinking": "off", "backend": "gpu"}) + "\n")
+    "host": "fixture", "verb": os.environ.get("ROW_VERB", "run"), "thinking": "off", "backend": "gpu"}) + "\n")
 PY
 }
 tokrow() { # tokrow <manifest> <dir> <pid> <ids csv>
@@ -384,6 +384,38 @@ expect "apr's degenerate '!!!!' is no answer: RED where llama.cpp answered" "$d"
 got=$(python3 -c 'import json,sys; c=[x for x in json.load(open(sys.argv[1]))["cells"] if x["key"]["prompt_id"]=="golden-greeting"][0]; print(c["engines"]["hf"]["answered"], c["engines"]["hf"]["why"][:18])' "$d/receipt.json" 2>/dev/null)
 case "$got" in "False degenerate output"*) ok "hf's degenerate '!!!!' cannot vouch either" ;; *) broke "hf degenerate: '$got'" ;; esac
 
+# C1-C3. the chat verb (#3739 slice 3): judged on the FINAL turn; apr's backend is
+#        recorded as unverified until apr chat reports one (#3794).
+apr_chat_out() { # apr_chat_out <dir> <pid> <reply 1> <reply 2> — apr chat's transcript shape
+  printf 'Detected ChatML chat template\nYou: [3.9s, ~2 tok/s]\nAssistant: %s\nYou: [5.9s, ~1 tok/s]\nAssistant: %s\nYou: \nGoodbye!\n' "$3" "$4" > "$1/apr-$2.out"
+  : > "$1/apr-$2.err"
+}
+chat_json() { # chat_json <dir> <engine> <pid> <reply 1> <reply 2> — the pty helper's JSON
+  python3 -c 'import json,sys; json.dump({"text": sys.argv[3], "turns": [sys.argv[2], sys.argv[3]], "reported": {"device": "cuda (-ngl 999)", "interface": "pty"}}, open(sys.argv[1], "w"))' \
+    "$1/$2-$3.json" "$4" "$5"
+}
+CP=chat-arith-2turn
+d=$(newcase chat_green); control_green "$d"
+apr_chat_out "$d" $CP "2 + 2 equals 4." "4 * 3 equals 12."; chat_json "$d" llama $CP "4" "12"
+ROW_VERB=chat row "$d/manifest.jsonl" apr $CP 0 "$d/apr-$CP.out" "$d/apr-$CP.err"
+ROW_VERB=chat row "$d/manifest.jsonl" llama.cpp $CP 0 "$d/llama-$CP.json" ""
+run_judge "$d"; GOT_RC=$?
+expect "a chat whose final turn carries the first turn's answer is GREEN" "$d" 0 $CP GREEN
+got=$(python3 -c 'import json,sys; c=[x for x in json.load(open(sys.argv[1]))["cells"] if x["key"]["verb"]=="chat"][0]; a=c["engines"]["apr"]; print(a["turns"], a["backend_verified"], c["token_parity"].get("why"))' "$d/receipt.json" 2>/dev/null)
+[ "$got" = "['2 + 2 equals 4.', '4 * 3 equals 12.'] False not measured for the chat verb" ] && ok "every apr turn recorded; backend unverified (#3794); no run-verb token parity" || broke "chat record: '$got'"
+d=$(newcase chat_red); control_green "$d"
+apr_chat_out "$d" $CP "2 + 2 equals 4." "4 * 3 equals 9."; chat_json "$d" llama $CP "4" "12"
+ROW_VERB=chat row "$d/manifest.jsonl" apr $CP 0 "$d/apr-$CP.out" "$d/apr-$CP.err"
+ROW_VERB=chat row "$d/manifest.jsonl" llama.cpp $CP 0 "$d/llama-$CP.json" ""
+run_judge "$d"; GOT_RC=$?
+expect "apr losing the first turn's answer while llama.cpp carries it is RED" "$d" 1 $CP RED
+d=$(newcase chat_no_turn); control_green "$d"
+printf 'Detected ChatML chat template\nYou: \nGoodbye!\n' > "$d/apr-$CP.out"; : > "$d/apr-$CP.err"; chat_json "$d" llama $CP "4" "12"
+ROW_VERB=chat row "$d/manifest.jsonl" apr $CP 0 "$d/apr-$CP.out" "$d/apr-$CP.err"
+ROW_VERB=chat row "$d/manifest.jsonl" llama.cpp $CP 0 "$d/llama-$CP.json" ""
+run_judge "$d"; GOT_RC=$?
+expect "an apr chat transcript with no Assistant turn is no answer (RED)" "$d" 1 $CP RED
+
 # D1-D4. tok: raw-text ids, BYTE-EQUAL or RED.
 d=$(newcase tok_equal); control_green "$d"
 printf '{"tokens": [9707, 11, 1879]}\n' > "$d/apr-tok.json"; printf '{"tokens": [9707, 11, 1879]}\n' > "$d/hf-tok.json"