diff --git a/docs/roadmaps/entries/PMAT-3795.yaml b/docs/roadmaps/entries/PMAT-3795.yaml new file mode 100644 index 0000000000..82deaaaabe --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3795.yaml @@ -0,0 +1,22 @@ +- id: PMAT-3795 + github_issue: 3795 + item_type: task + title: 'CRUX slice 3 (#3739): the chat verb — a two-turn conversation through apr chat, llama.cpp and ollama (pty-driven), hf and llamafile; judged on the final turn' + status: in_progress + priority: critical + assigned_to: aprender-76 + created: 2026-09-21T23:00:00Z + updated: 2026-09-21T23:20:00Z + spec: null + acceptance_criteria: + - 'Issue #3795 done_when 1, verbatim: "`scripts/crux_inference_dogfood.sh --verbs run,chat` runs a multi-turn conversation per model through every engine. A `chat` request without `run` declines, because the positive control lives in `run`." The prompt is `chat-arith-2turn`, the user turns "What is 2+2?" then "Now multiply that by 3.", judged on the FINAL reply (`expect_any ["12"]`); every reply is recorded.' + - 'done_when 2, verbatim: apr runs through `apr chat`, one user turn per stdin line; llama.cpp and ollama through `scripts/lib/crux_pty_chat.py`, a stdlib pseudo-terminal driver, because they do not take turns from a pipe. MEASURED before building it: llama-cli fed a pipe loops on empty `> ` prompts until killed; `ollama run` reads the whole pipe as ONE prompt and answers both turns in one reply. Driven through the pty helper, each held the conversation: llama-cli and ollama both gave ["2 + 2 equals 4.", "4 * 3 equals 12."]. hf and llamafile use `gen --verb chat` with the user turns (infra-3c, a3eecfe2d).' + - 'done_when 3, verbatim: "The judge scores the final turn with the same RED rule. `apr chat` reports no backend (#3794), so apr''s chat cells carry `backend_verified: false` and the receipt''s `not_covered` names it. The run-verb token-parity field isn''t reported as measured for chat."' + - 'done_when 4, MEASURED: `bash scripts/check_crux_inference_judge.sh` gives 42 ok, 0 broke. The chat rows: a GREEN chat; apr losing the first turn''s answer ("9") while llama.cpp carries it ("12") is RED; an apr transcript with no `Assistant:` turn is RED; and the record of every turn, `backend_verified: false`, and "not measured for the chat verb". Mutants: judging apr chat on the FIRST turn breaks 1; not recording backend_verified breaks 1; measuring chat as run breaks 1.' + - 'done_when 5, MEASURED on lambda (cpu lane, qwen2.5-coder-1.5b Q4_K_M, `--verbs run,chat`): PASS, 4 cells (3 run + 1 chat), 4 GREEN. In the chat cell apr, llama.cpp@d1d3c3396 and ollama 0.33.2 each gave ["2 + 2 equals 4.", "4 * 3 equals 12."]. hf and llamafile were refused by name (their drivers are on PMAT-3778, not yet merged). `not_covered` names apr chat''s unverified backend (#3794). No ollama import was left behind.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'Refs #3739 (slice 3; not a closing keyword for the parent). Closes #3795 when this merges. STACKED on crux-3774-verb-correspondence (#3789), so the diff to judge is `origin/crux-3774-verb-correspondence...HEAD`. SCOPE: scripts/crux_inference_dogfood.sh (the chat verb), scripts/lib/crux_pty_chat.py (new), scripts/lib/crux_inference_judge.py (chat parsing), scripts/check_crux_inference_judge.sh (chat rows), scripts/crux_inference_prompts.json (the chat prompt; the golden drift check covers only the golden rung), and this fragment. FOUND while building it: #3794, where `apr chat --no-gpu` still uploads the GGUF to CUDA (VRAM held outside the GPU lock; generation honours force_cpu, the load path does not) and apr chat reports no machine-readable backend. The pty helper starts only the argv it is given and reads no clock for a metric (its per-turn timeout is a bound), so the competing-harness count stays 0.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 1d0b3940ad..65bc185996 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -20021,3 +20021,25 @@ roadmap: labels: - kind:code notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml, evidence/crux/hf-sources.yaml, scripts/lib/crux_inference_judge.py, scripts/crux_inference_dogfood.sh (the plugin-engine hook), scripts/check_crux_inference_judge.sh, and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' +- id: PMAT-3795 + github_issue: 3795 + item_type: task + title: 'CRUX slice 3 (#3739): the chat verb — a two-turn conversation through apr chat, llama.cpp and ollama (pty-driven), hf and llamafile; judged on the final turn' + status: in_progress + priority: critical + assigned_to: aprender-76 + created: 2026-09-21T23:00:00Z + updated: 2026-09-21T23:20:00Z + spec: null + acceptance_criteria: + - 'Issue #3795 done_when 1, verbatim: "`scripts/crux_inference_dogfood.sh --verbs run,chat` runs a multi-turn conversation per model through every engine. A `chat` request without `run` declines, because the positive control lives in `run`." The prompt is `chat-arith-2turn`, the user turns "What is 2+2?" then "Now multiply that by 3.", judged on the FINAL reply (`expect_any ["12"]`); every reply is recorded.' + - 'done_when 2, verbatim: apr runs through `apr chat`, one user turn per stdin line; llama.cpp and ollama through `scripts/lib/crux_pty_chat.py`, a stdlib pseudo-terminal driver, because they do not take turns from a pipe. MEASURED before building it: llama-cli fed a pipe loops on empty `> ` prompts until killed; `ollama run` reads the whole pipe as ONE prompt and answers both turns in one reply. Driven through the pty helper, each held the conversation: llama-cli and ollama both gave ["2 + 2 equals 4.", "4 * 3 equals 12."]. hf and llamafile use `gen --verb chat` with the user turns (infra-3c, a3eecfe2d).' + - 'done_when 3, verbatim: "The judge scores the final turn with the same RED rule. `apr chat` reports no backend (#3794), so apr''s chat cells carry `backend_verified: false` and the receipt''s `not_covered` names it. The run-verb token-parity field isn''t reported as measured for chat."' + - 'done_when 4, MEASURED: `bash scripts/check_crux_inference_judge.sh` gives 42 ok, 0 broke. The chat rows: a GREEN chat; apr losing the first turn''s answer ("9") while llama.cpp carries it ("12") is RED; an apr transcript with no `Assistant:` turn is RED; and the record of every turn, `backend_verified: false`, and "not measured for the chat verb". Mutants: judging apr chat on the FIRST turn breaks 1; not recording backend_verified breaks 1; measuring chat as run breaks 1.' + - 'done_when 5, MEASURED on lambda (cpu lane, qwen2.5-coder-1.5b Q4_K_M, `--verbs run,chat`): PASS, 4 cells (3 run + 1 chat), 4 GREEN. In the chat cell apr, llama.cpp@d1d3c3396 and ollama 0.33.2 each gave ["2 + 2 equals 4.", "4 * 3 equals 12."]. hf and llamafile were refused by name (their drivers are on PMAT-3778, not yet merged). `not_covered` names apr chat''s unverified backend (#3794). No ollama import was left behind.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'Refs #3739 (slice 3; not a closing keyword for the parent). Closes #3795 when this merges. STACKED on crux-3774-verb-correspondence (#3789), so the diff to judge is `origin/crux-3774-verb-correspondence...HEAD`. SCOPE: scripts/crux_inference_dogfood.sh (the chat verb), scripts/lib/crux_pty_chat.py (new), scripts/lib/crux_inference_judge.py (chat parsing), scripts/check_crux_inference_judge.sh (chat rows), scripts/crux_inference_prompts.json (the chat prompt; the golden drift check covers only the golden rung), and this fragment. FOUND while building it: #3794, where `apr chat --no-gpu` still uploads the GGUF to CUDA (VRAM held outside the GPU lock; generation honours force_cpu, the load path does not) and apr chat reports no machine-readable backend. The pty helper starts only the argv it is given and reads no clock for a metric (its per-turn timeout is a bound), so the competing-harness count stays 0.' diff --git a/scripts/check_crux_inference_judge.sh b/scripts/check_crux_inference_judge.sh index 17c67880b7..4655fadc5d 100755 --- a/scripts/check_crux_inference_judge.sh +++ b/scripts/check_crux_inference_judge.sh @@ -63,12 +63,12 @@ ollama_out() { # ollama_out } row() { # row [refused] python3 - "$@" <<'PY' -import json, sys +import json, os, sys m, eng, pid, rc, o, e = sys.argv[1:7] ref = sys.argv[7] if len(sys.argv) > 7 else "" open(m, "a").write(json.dumps({"kind": "gen", "engine": eng, "prompt_id": pid, "rc": int(rc) if rc else None, "stdout": o or None, "stderr": e or None, "refused": ref or None, "model_sha256": "%s", - "host": "fixture", "verb": "run", "thinking": "off", "backend": "gpu"}) + "\n") + "host": "fixture", "verb": os.environ.get("ROW_VERB", "run"), "thinking": "off", "backend": "gpu"}) + "\n") PY } tokrow() { # tokrow @@ -384,6 +384,38 @@ expect "apr's degenerate '!!!!' is no answer: RED where llama.cpp answered" "$d" got=$(python3 -c 'import json,sys; c=[x for x in json.load(open(sys.argv[1]))["cells"] if x["key"]["prompt_id"]=="golden-greeting"][0]; print(c["engines"]["hf"]["answered"], c["engines"]["hf"]["why"][:18])' "$d/receipt.json" 2>/dev/null) case "$got" in "False degenerate output"*) ok "hf's degenerate '!!!!' cannot vouch either" ;; *) broke "hf degenerate: '$got'" ;; esac +# C1-C3. the chat verb (#3739 slice 3): judged on the FINAL turn; apr's backend is +# recorded as unverified until apr chat reports one (#3794). +apr_chat_out() { # apr_chat_out — apr chat's transcript shape + printf 'Detected ChatML chat template\nYou: [3.9s, ~2 tok/s]\nAssistant: %s\nYou: [5.9s, ~1 tok/s]\nAssistant: %s\nYou: \nGoodbye!\n' "$3" "$4" > "$1/apr-$2.out" + : > "$1/apr-$2.err" +} +chat_json() { # chat_json — the pty helper's JSON + python3 -c 'import json,sys; json.dump({"text": sys.argv[3], "turns": [sys.argv[2], sys.argv[3]], "reported": {"device": "cuda (-ngl 999)", "interface": "pty"}}, open(sys.argv[1], "w"))' \ + "$1/$2-$3.json" "$4" "$5" +} +CP=chat-arith-2turn +d=$(newcase chat_green); control_green "$d" +apr_chat_out "$d" $CP "2 + 2 equals 4." "4 * 3 equals 12."; chat_json "$d" llama $CP "4" "12" +ROW_VERB=chat row "$d/manifest.jsonl" apr $CP 0 "$d/apr-$CP.out" "$d/apr-$CP.err" +ROW_VERB=chat row "$d/manifest.jsonl" llama.cpp $CP 0 "$d/llama-$CP.json" "" +run_judge "$d"; GOT_RC=$? +expect "a chat whose final turn carries the first turn's answer is GREEN" "$d" 0 $CP GREEN +got=$(python3 -c 'import json,sys; c=[x for x in json.load(open(sys.argv[1]))["cells"] if x["key"]["verb"]=="chat"][0]; a=c["engines"]["apr"]; print(a["turns"], a["backend_verified"], c["token_parity"].get("why"))' "$d/receipt.json" 2>/dev/null) +[ "$got" = "['2 + 2 equals 4.', '4 * 3 equals 12.'] False not measured for the chat verb" ] && ok "every apr turn recorded; backend unverified (#3794); no run-verb token parity" || broke "chat record: '$got'" +d=$(newcase chat_red); control_green "$d" +apr_chat_out "$d" $CP "2 + 2 equals 4." "4 * 3 equals 9."; chat_json "$d" llama $CP "4" "12" +ROW_VERB=chat row "$d/manifest.jsonl" apr $CP 0 "$d/apr-$CP.out" "$d/apr-$CP.err" +ROW_VERB=chat row "$d/manifest.jsonl" llama.cpp $CP 0 "$d/llama-$CP.json" "" +run_judge "$d"; GOT_RC=$? +expect "apr losing the first turn's answer while llama.cpp carries it is RED" "$d" 1 $CP RED +d=$(newcase chat_no_turn); control_green "$d" +printf 'Detected ChatML chat template\nYou: \nGoodbye!\n' > "$d/apr-$CP.out"; : > "$d/apr-$CP.err"; chat_json "$d" llama $CP "4" "12" +ROW_VERB=chat row "$d/manifest.jsonl" apr $CP 0 "$d/apr-$CP.out" "$d/apr-$CP.err" +ROW_VERB=chat row "$d/manifest.jsonl" llama.cpp $CP 0 "$d/llama-$CP.json" "" +run_judge "$d"; GOT_RC=$? +expect "an apr chat transcript with no Assistant turn is no answer (RED)" "$d" 1 $CP RED + # D1-D4. tok: raw-text ids, BYTE-EQUAL or RED. d=$(newcase tok_equal); control_green "$d" printf '{"tokens": [9707, 11, 1879]}\n' > "$d/apr-tok.json"; printf '{"tokens": [9707, 11, 1879]}\n' > "$d/hf-tok.json" diff --git a/scripts/crux_inference_dogfood.sh b/scripts/crux_inference_dogfood.sh index 6972e2c80f..f2f48b840e 100755 --- a/scripts/crux_inference_dogfood.sh +++ b/scripts/crux_inference_dogfood.sh @@ -100,8 +100,9 @@ done [ "${#MODELS[@]}" -gt 0 ] || decline "no --model given; this slice takes models by path" case "$BACKEND" in gpu|cpu) ;; *) decline "--backend is gpu or cpu, got '$BACKEND'" ;; esac case ",$VERBS," in - ,run,) ;; - *) decline "verbs '$VERBS': this slice drives 'run' only; chat/serve/code are the next slices of #3739 and are listed as not covered in every receipt" ;; + ,run,|,run,chat,|,chat,run,) ;; + *,chat,*) decline "verbs '$VERBS': chat needs run beside it (--verbs run,chat), because the positive control lives in the run verb" ;; + *) decline "verbs '$VERBS': this driver runs 'run' and 'chat'; serve and code are later slices of #3739 and are listed as not covered in every receipt" ;; esac want() { case ",$ENGINES," in *",$1,"*) return 0 ;; *) return 1 ;; esac; } want apr || decline "apr is the subject; --engines must include it" @@ -241,15 +242,23 @@ export CRUX_MANIFEST="$MANIFEST" CRUX_WORK="$WORK" MODELS_JSONL="$WORK/models.jsonl"; : > "$MODELS_JSONL" # One file per prompt: its id list, and each prompt's last user message. -PIDS=$(python3 - "$PROMPTS" "$WORK" <<'PY' +# A prompt belongs to one verb ("verb", default run). A chat prompt's messages are +# the USER turns of one conversation; it is judged on the final turn. +PIDS_ALL=$(python3 - "$PROMPTS" "$WORK" <<'PY' import json, sys d = json.load(open(sys.argv[1])) for p in d["prompts"]: - open("%s/prompt-%s.txt" % (sys.argv[2], p["id"]), "w").write(p["messages"][-1]["content"]) - json.dump({"messages": p["messages"]}, open("%s/messages-%s.json" % (sys.argv[2], p["id"]), "w")) - print(p["id"]) + w = sys.argv[2] + open("%s/prompt-%s.txt" % (w, p["id"]), "w").write(p["messages"][-1]["content"]) + json.dump({"messages": p["messages"]}, open("%s/messages-%s.json" % (w, p["id"]), "w")) + users = [m["content"] for m in p["messages"] if m.get("role") == "user"] + json.dump(users, open("%s/turns-%s.json" % (w, p["id"]), "w")) + open("%s/turns-%s.txt" % (w, p["id"]), "w").write("".join(u + "\n" for u in users)) + print("%s %s" % (p.get("verb", "run"), p["id"])) PY ) || decline "prompt set $PROMPTS unreadable" +pids_for() { printf '%s\n' "$PIDS_ALL" | sed -n "s/^$1 //p" | tr '\n' ' '; } +PIDS=$(pids_for run) MAXTOK=$(python3 -c 'import json,sys; print(int(json.load(open(sys.argv[1]))["max_tokens"]))' "$PROMPTS") || decline "max_tokens unreadable" # ONE CELL = ONE COMMAND UNDER THE GPU LOCK (the cop's rule rev 5, #3739). A cell is @@ -293,13 +302,18 @@ run_cell() { # run_cell ; sets CELL_WHY when the lock was not had fi return "$rc" } -cell_result() { # cell_result — the engine's row from its cell files +cell_result() { # cell_result [stdout file] — the engine's row from its cell files local rc unl="" if [ -n "$CELL_WHY" ]; then emit_gen "$1" "$2" "" "" "" "$CELL_WHY"; return; fi if [ ! -f "$3.rc" ]; then emit_gen "$1" "$2" "" "" "" "the cell ran but this engine's line never finished"; return; fi rc=$(cat "$3.rc") [ -f "$3.unloaded" ] && unl=$(cat "$3.unloaded") - emit_gen "$1" "$2" "$rc" "$3.out" "$3.err" "" "$unl" + emit_gen "$1" "$2" "$rc" "${4:-$3.out}" "$3.err" "" "$unl" +} +cell_add_stdin() { # cell_add_stdin cmd... — like cell_add, fed a file + local cell="$1" prefix="$2" input="$3"; shift 3 + { printf 'timeout %q ' "$TMO"; printf '%q ' "$@" + printf '> %q 2> %q < %q; echo $? > %q\n' "$prefix.out" "$prefix.err" "$input" "$prefix.rc"; } >> "$cell" } emit_gen() { # emit_gen [ollama_unloaded] @@ -384,6 +398,12 @@ case "$BACKEND" in gpu) APR_BE="--gpu"; NGL=999; LLAMA_DEV=() ;; cpu) APR_BE="--no-gpu"; NGL=0; LLAMA_DEV=(-dev none) ;; esac +# The device the chat cells' pty-driven comparators were TOLD to use: their own +# flags, recorded in the row (these engines are not held to a measured device). +case "$BACKEND" in + gpu) LLAMA_DEVICE="cuda (-ngl 999)"; OL_DEVICE="gpu (ollama's default placement)" ;; + cpu) LLAMA_DEVICE="cpu (-ngl 0 -dev none)"; OL_DEVICE="cpu (PARAMETER num_gpu 0)" ;; +esac VERB=run for M_IN in "${MODELS[@]}"; do M=$(readlink -f "$M_IN" 2>/dev/null) || M="" @@ -477,21 +497,43 @@ PY fi fi - for pid in $PIDS; do + for VERB in ${VERBS//,/ }; do + for pid in $(pids_for "$VERB"); do content=$(cat "$WORK/prompt-$pid.txt") d="$WORK/$SHA12/$VERB" mkdir -p "$d" cell="$d/cell-$pid.sh" - printf '#!/usr/bin/env bash\n# one CRUX cell: prompt %s through every engine, one hold of the GPU lock\n' "$pid" > "$cell" + printf '#!/usr/bin/env bash\n# one CRUX cell: %s prompt %s through every engine, one hold of the GPU lock\n' "$VERB" "$pid" > "$cell" - cell_add "$cell" "$d/apr-$pid" "$APR" run "$M" --prompt "$content" --max-tokens "$MAXTOK" \ - --temperature "$TEMP" --seed "$SEED" --format json -v "$APR_BE" - [ "$LLAMA_OK" = 1 ] && cell_add "$cell" "$d/llama-$pid" "$LLAMA_CLI" -m "$M" -p "$content" -st -n "$MAXTOK" \ - --temp "$TEMP" --seed "$SEED" -c "$CTX" -ngl "$NGL" "${LLAMA_DEV[@]}" "${LLAMA_THINK[@]}" - if [ "$OLLAMA_OK" = 1 ] && [ -z "$OL_REFUSED" ]; then - cell_add "$cell" "$d/ollama-$pid" "$OLLAMA" run "$OL_NAME" "$content" --verbose --nowordwrap \ - --keepalive 0 "${OLLAMA_THINK[@]}" - cell_add_ollama_unload "$cell" "$d/ollama-$pid" "$OL_NAME" + if [ "$VERB" = chat ]; then + # apr chat reads one user turn per stdin line. The competitors' chat CLIs + # do not take turns from a pipe (measured: llama.cpp loops on empty + # prompts, ollama reads the pipe as ONE prompt), so scripts/lib/ + # crux_pty_chat.py drives them through a pseudo-terminal, one turn at a + # time, and writes the row-contract JSON the judge reads. + cell_add_stdin "$cell" "$d/apr-$pid" "$WORK/turns-$pid.txt" "$APR" chat "$M" \ + --temperature "$TEMP" --max-tokens "$MAXTOK" "$APR_BE" + [ "$LLAMA_OK" = 1 ] && cell_add "$cell" "$d/llama-$pid" python3 scripts/lib/crux_pty_chat.py \ + --marker '(?m)^> $' --turns "$WORK/turns-$pid.json" --out "$d/llama-$pid.json" --exit-line /exit \ + --device "$LLAMA_DEVICE" --turn-timeout "$TMO" --start-timeout "$TMO" --strip '\[ Prompt:[^]]*\]' -- \ + "$LLAMA_CLI" -m "$M" -n "$MAXTOK" --temp "$TEMP" --seed "$SEED" -c "$CTX" -ngl "$NGL" "${LLAMA_DEV[@]}" "${LLAMA_THINK[@]}" + if [ "$OLLAMA_OK" = 1 ] && [ -z "$OL_REFUSED" ]; then + cell_add "$cell" "$d/ollama-$pid" python3 scripts/lib/crux_pty_chat.py \ + --marker '>>> ' --answer-after '^[.][.][.].*$' --turns "$WORK/turns-$pid.json" --out "$d/ollama-$pid.json" \ + --exit-line /bye --device "$OL_DEVICE" --turn-timeout "$TMO" --start-timeout "$TMO" -- \ + "$OLLAMA" run "$OL_NAME" --keepalive 0 --nowordwrap "${OLLAMA_THINK[@]}" + cell_add_ollama_unload "$cell" "$d/ollama-$pid" "$OL_NAME" + fi + else + cell_add "$cell" "$d/apr-$pid" "$APR" run "$M" --prompt "$content" --max-tokens "$MAXTOK" \ + --temperature "$TEMP" --seed "$SEED" --format json -v "$APR_BE" + [ "$LLAMA_OK" = 1 ] && cell_add "$cell" "$d/llama-$pid" "$LLAMA_CLI" -m "$M" -p "$content" -st -n "$MAXTOK" \ + --temp "$TEMP" --seed "$SEED" -c "$CTX" -ngl "$NGL" "${LLAMA_DEV[@]}" "${LLAMA_THINK[@]}" + if [ "$OLLAMA_OK" = 1 ] && [ -z "$OL_REFUSED" ]; then + cell_add "$cell" "$d/ollama-$pid" "$OLLAMA" run "$OL_NAME" "$content" --verbose --nowordwrap \ + --keepalive 0 "${OLLAMA_THINK[@]}" + cell_add_ollama_unload "$cell" "$d/ollama-$pid" "$OL_NAME" + fi fi for eng in hf llamafile; do want "$eng" && [ "${EXT_OK[$eng]}" = 1 ] || continue @@ -513,10 +555,11 @@ PY before_hf=$(rows_for hf "$pid"); before_lf=$(rows_for llamafile "$pid") run_cell "$cell" + pty_out=""; [ "$VERB" = chat ] && pty_out=json cell_result apr "$pid" "$d/apr-$pid" - if [ "$LLAMA_OK" = 1 ]; then cell_result llama.cpp "$pid" "$d/llama-$pid" + if [ "$LLAMA_OK" = 1 ]; then cell_result llama.cpp "$pid" "$d/llama-$pid" "${pty_out:+$d/llama-$pid.json}" elif want llama.cpp; then emit_gen llama.cpp "$pid" "" "" "" "$LLAMA_WHY"; fi - if [ "$OLLAMA_OK" = 1 ] && [ -z "$OL_REFUSED" ]; then cell_result ollama "$pid" "$d/ollama-$pid" + if [ "$OLLAMA_OK" = 1 ] && [ -z "$OL_REFUSED" ]; then cell_result ollama "$pid" "$d/ollama-$pid" "${pty_out:+$d/ollama-$pid.json}" elif want ollama; then emit_gen ollama "$pid" "" "" "" "${OL_REFUSED:-$OLLAMA_WHY}"; fi for eng in hf llamafile; do want "$eng" || continue @@ -529,6 +572,7 @@ PY fi done done + done if [ "$OLLAMA_OK" = 1 ] && [ "$KEEP_OLLAMA" = 0 ]; then "$OLLAMA" rm "$OL_NAME" > /dev/null 2>&1 @@ -563,7 +607,9 @@ meta = { **json.load(open(ext)), "models": [json.loads(l) for l in open(models) if l.strip()], "not_covered": [ - "verbs chat, serve and code (the next slices of #3739)", + ("verbs serve and code (later slices of #3739)" if "chat" in verbs.split(",") + else "verbs chat, serve and code (later slices of #3739)"), + *(["apr chat's backend is unverified: apr chat reports none (#3794)"] if "chat" in verbs.split(",") else []), "thinking ON (apr has no toggle until #3723)", "consumer-brief context rungs (#3716) and each engine's max accepted context", "TTFT, which belongs to the serve verb's single OpenAI client", diff --git a/scripts/crux_inference_prompts.json b/scripts/crux_inference_prompts.json index c8fdc73f78..4412d464de 100644 --- a/scripts/crux_inference_prompts.json +++ b/scripts/crux_inference_prompts.json @@ -1,6 +1,7 @@ { "schema": "crux-inference-prompts/v1", "source": "crates/apr-cli/src/commands/golden_output.rs::golden_test_cases, the same three cases given as USER MESSAGES rather than pre-rendered ChatML: each engine applies the chat template it reads from the GGUF, and that step is part of what is under test (#3672 is a template defect a pre-rendered prompt would hide). scripts/check_crux_inference_judge.sh refuses drift between the two lists.", + "chat": "chat-arith-2turn is the chat verb's conversation: two USER turns, judged on the FINAL reply (12 needs the first turn's 4 carried over). Every reply is recorded. Prompts without a \"verb\" belong to run.", "control": "golden-2plus2 is the POSITIVE CONTROL (the cop's ruling on #3739): the smallest model answers it right on every engine, so a control cell that comes back ALL_WRONG declines the run as a broken harness rather than recording a model limitation.", "max_tokens": 64, "prompts": [ @@ -22,6 +23,13 @@ "rung": "golden", "messages": [{"role": "user", "content": "What is the capital of France?"}], "expect_any": ["Paris"] + }, + { + "id": "chat-arith-2turn", + "rung": "chat", + "verb": "chat", + "messages": [{"role": "user", "content": "What is 2+2?"}, {"role": "user", "content": "Now multiply that by 3."}], + "expect_any": ["12"] } ] } diff --git a/scripts/lib/crux_inference_judge.py b/scripts/lib/crux_inference_judge.py index e2aee299e2..abef8a2b62 100644 --- a/scripts/lib/crux_inference_judge.py +++ b/scripts/lib/crux_inference_judge.py @@ -175,6 +175,22 @@ def num(key): return out +def parse_apr_chat(stdout): + """apr's `chat` verb, fed one user turn per stdin line: each reply follows an + `Assistant: ` prefix and runs to the next `You:` prompt. The final reply is + the answer; every reply is recorded.""" + out = {"answer": None, "why": None, "reported": {"reported_by": "apr"}, "turns": []} + text = ANSI.sub("", stdout) + for seg in re.split(r"(?m)^Assistant: ", text)[1:]: + cut = re.search(r"(?m)^You:", seg) + out["turns"].append((seg[:cut.start()] if cut else seg).strip()) + if not out["turns"]: + out["why"] = "no `Assistant:` turn in the transcript" + return out + out["answer"] = out["turns"][-1] + return out + + def parse_engine_json(stdout): """hf and llamafile rows (row contract v1, #3739 issuecomment-5765991210): the engine driver writes `{"text": , "reported": {...}}`.""" @@ -190,6 +206,8 @@ def parse_engine_json(stdout): rep = doc.get("reported") out["reported"] = rep if isinstance(rep, dict) else {} out["answer"] = doc["text"] + if isinstance(doc.get("turns"), list): + out["turns"] = doc["turns"] return out @@ -227,7 +245,20 @@ def engine_entry(row, prompt): stdout, stderr = read_text(row.get("stdout")), read_text(row.get("stderr")) content = prompt["messages"][-1]["content"] engine = row["engine"] - if engine == "apr": + if row.get("verb") == "chat": + # chat (#3739 slice 3): apr's own transcript; every other engine writes the + # row-contract JSON (the pty helper for llama.cpp/ollama, the plugin drivers). + if engine == "apr": + p = parse_apr_chat(stdout) + # apr chat reports no backend (#3794): recorded, not scored as verified. + e["backend_verified"] = False + elif engine in COMPARATORS: + p = parse_engine_json(stdout) + else: + e["why"] = "unknown engine %r" % engine + return e + e["turns"] = p.get("turns") or [] + elif engine == "apr": p = parse_apr(stdout, stderr) e["backend"] = p["backend"] e["prompt_ids"] = p["prompt_ids"] @@ -505,7 +536,8 @@ def collect(args): "all_identical": len(said) >= 2 and len(set(said.values())) == 1, "apr_matches": sorted(e for e in said if e != "apr" and "apr" in said and said[e] == said["apr"]), }, - "token_parity": token_parity(entries["apr"], toks.get((k[0], k[5]))), + "token_parity": (token_parity(entries["apr"], toks.get((k[0], k[5]))) if k[2] == "run" + else {"measured": False, "why": "not measured for the %s verb" % k[2]}), }) counts = {v: sum(1 for c in cells if c["verdict"] == v) for v in ("RED", "GREEN", "UNJUDGED", "ALL_WRONG")} diff --git a/scripts/lib/crux_pty_chat.py b/scripts/lib/crux_pty_chat.py new file mode 100755 index 0000000000..745a78abc1 --- /dev/null +++ b/scripts/lib/crux_pty_chat.py @@ -0,0 +1,159 @@ +#!/usr/bin/env python3 +"""crux_pty_chat.py: drive an interactive chat CLI through a pseudo-terminal (#3739, the `chat` verb). + +WHY. Measured on lambda: fed a pipe, llama.cpp's chat CLI loops on empty prompts +and ollama's interactive mode reads the whole pipe as ONE prompt. Neither takes a +multi-turn conversation on stdin, so they are driven the way a person drives them: +wait for the prompt marker, type one turn, capture the reply up to the next marker. + +It starts nothing but the argv it is given, opens no socket and reads no clock +for any measurement (the per-turn timeout is a bound, not a metric). Output is +the row-contract JSON the judge already reads for plugin engines: + {"text": , "turns": [], "reported": {"device": ...}} +Exit: 0 every turn answered · 3 a turn timed out or the process died (the JSON +still says which turn and why) · 2 usage. + +Usage: crux_pty_chat.py --marker REGEX --turns FILE.json --out FILE.json + [--exit-line TEXT] [--device LABEL] [--turn-timeout S] [--start-timeout S] + [--strip REGEX]... [--answer-after REGEX] -- CMD [ARGS...] +""" +import argparse +import json +import os +import pty +import re +import select +import signal +import sys +import time + +ANSI = re.compile(r"\x1b\[[0-9;?]*[ -/]*[@-~]|\x1b\][^\x07]*\x07|\x1b[()][AB012]|\r") + + +def clean(text): + return ANSI.sub("", text) + + +class Session: + def __init__(self, argv): + self.pid, self.fd = pty.fork() + if self.pid == 0: + os.environ["TERM"] = "dumb" + try: + os.execvp(argv[0], argv) + finally: + os._exit(127) + self.buf = "" + + def read_until(self, marker, since, bound): + """Read until `marker` matches the cleaned output after offset `since`.""" + deadline = time.monotonic() + bound + while True: + m = marker.search(clean(self.buf)[since:]) + if m: + return since + m.start() + left = deadline - time.monotonic() + if left <= 0: + return None + r, _, _ = select.select([self.fd], [], [], min(left, 0.5)) + if r: + try: + chunk = os.read(self.fd, 65536) + except OSError: + return None + if not chunk: + return None + self.buf += chunk.decode("utf-8", "replace") + + def send(self, line): + os.write(self.fd, (line + "\r").encode("utf-8")) + + def close(self, exit_line): + if exit_line: + try: + self.send(exit_line) + except OSError: + pass + for _ in range(20): + done, _ = os.waitpid(self.pid, os.WNOHANG) + if done: + return + time.sleep(0.25) + try: + os.kill(self.pid, signal.SIGTERM) + time.sleep(1) + os.kill(self.pid, signal.SIGKILL) + except OSError: + pass + try: + os.waitpid(self.pid, 0) + except OSError: + pass + + +def main(argv): + if "--" not in argv: + sys.stderr.write("usage: crux_pty_chat.py [options] -- CMD [ARGS...]\n") + return 2 + cut = argv.index("--") + ap = argparse.ArgumentParser() + ap.add_argument("--marker", required=True) + ap.add_argument("--turns", required=True) + ap.add_argument("--out", required=True) + ap.add_argument("--exit-line", default="") + ap.add_argument("--device", default="") + ap.add_argument("--turn-timeout", type=float, default=300) + ap.add_argument("--start-timeout", type=float, default=300) + ap.add_argument("--strip", action="append", default=[]) + ap.add_argument("--answer-after", default="", + help="the reply starts after the LAST match of this regex (an echo the terminal breaks up)") + a = ap.parse_args(argv[:cut]) + cmd = argv[cut + 1:] + if not cmd: + sys.stderr.write("no command after --\n") + return 2 + turns = json.load(open(a.turns)) + marker = re.compile(a.marker) + strips = [re.compile(s) for s in a.strip] + s = Session(cmd) + out = {"text": None, "turns": [], "reported": {"device": a.device or None, "interface": "pty"}} + rc = 0 + at = s.read_until(marker, 0, a.start_timeout) + if at is None: + out["error"] = "no prompt marker within %.0fs of start" % a.start_timeout + rc = 3 + else: + pos = at + 1 + for i, turn in enumerate(turns): + s.send(turn) + end = s.read_until(marker, pos, a.turn_timeout) + if end is None: + out["error"] = "turn %d: no prompt marker within %.0fs" % (i + 1, a.turn_timeout) + rc = 3 + break + reply = clean(s.buf)[pos:end] + if a.answer_after: + last = None + for last in re.finditer(a.answer_after, reply, re.M): + pass + if last is not None: + reply = reply[last.end():] + else: + k = reply.find(turn) # the terminal echoes what was typed; the reply follows it + if k >= 0: + reply = reply[k + len(turn):] + for rx in strips: + reply = rx.sub("", reply) + out["turns"].append(reply.strip()) + pos = end + 1 + s.close(a.exit_line) + if out["turns"] and rc == 0: + out["text"] = out["turns"][-1] + out["transcript_tail"] = clean(s.buf)[-1500:] + with open(a.out, "w", encoding="utf-8") as fh: + json.dump(out, fh, ensure_ascii=False) + return rc + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:]))