diff --git a/docs/roadmaps/entries/PMAT-3739.yaml b/docs/roadmaps/entries/PMAT-3739.yaml new file mode 100644 index 0000000000..9b138e0dca --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3739.yaml @@ -0,0 +1,24 @@ +- id: PMAT-3739 + github_issue: 3739 + item_type: task + title: 'CRUX inference dogfood slice 1: the SAME GGUF + SAME golden prompts through apr, llama.cpp@pin and ollama on the `run` verb, judged by one rule: a competitor right and apr wrong is RED' + status: in_progress + priority: critical + assigned_to: aprender-76 + created: 2026-09-21T17:10:00Z + updated: 2026-09-21T20:20:00Z + spec: null + acceptance_criteria: + - 'SLICE 1 of #3739, as the design note on the issue lays out (issuecomment-5764437831): the `run` verb, the golden prompts, thinking OFF, lambda. This PR does NOT meet the issue''s done_when in full, and it says `Refs #3739`, never a closing keyword. Every receipt it writes carries a `not_covered` list naming the verbs chat, serve and code, thinking ON, the consumer-brief rungs and max context, and TTFT, so a partial run cannot be read as a full one.' + - 'done_when 1, the part this slice meets, verbatim: "`scripts/crux_inference_dogfood.sh ` runs ... the SAME prompt set (the golden prompts ...) through: run: `apr run` · `llama-cli` · `ollama run`". Measured: `bash scripts/crux_inference_dogfood.sh 0.69.0 --backend cpu --model ~/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf` on lambda gives rc 0 with 3 cells GREEN, apr and llama.cpp@d1d3c3396 both correct. It declines (rc 2) when `apr --version` is not , when no --model is given, for a verb other than run, when --engines omits apr, and when no comparator resolves.' + - '"Same model" is recorded, not asserted, verbatim: "ollama IMPORTS the exact GGUF via a Modelfile (`FROM `) and the receipt records sha256 for all three engines". The receipt holds the input sha256 and ollama''s stored blob digest, with `blob_is_input`. An ollama import that carries no chat template (`TEMPLATE {{ .Prompt }}`, what 0.5.7 does on lambda) is REFUSED by name for every cell, never run as a raw completion against two engines that chat. GPU sharing, per the cop''s ruling: on the gpu lane one cell (one prompt through every engine) holds /tmp/apr-gpu.lock (fd 9, closed in every engine) for its engine invocations only. ollama runs with `--keepalive 0`, and the cell stops the model and waits for `ollama ps` to drop it before releasing the lock, recording `vram_released`. The cpu lane takes no lock (the rule''s clause 2); every engine runs under `choom -n 1000`.' + - 'done_when 2, the part this slice meets: per cell the receipt records answer correct (golden), exact-match agreement across engines, token-id parity (apr''s `-v` prompt ids against llama.cpp''s own `/apply-template` + `/tokenize` rendering of the same messages; count plus the id prefix apr prints, labelled `apr_ids_complete`), and prompt/completion tokens and rates as each engine REPORTED them, labelled by engine and never judged. The driver reads no clock and computes no rate (`check_no_competing_harnesses.sh` stays count=0). TTFT and max context belong to later slices and are listed as not covered.' + - 'done_when 3, the gate, verbatim: "a cell where llama.cpp OR ollama answers correctly and apr doesn''t is **RED** (a finding naming the cell)". `bash scripts/check_crux_inference_judge.sh`, which guard_tree runs bare, gives rc 0 and "21 ok, 0 broke". Its row 2 is the #3715 falsifier: llama.cpp right and apr wrong makes the judge exit 1 with the cell RED. Other rows make RED an apr that exits non-zero, falls back off the lane''s backend, or is MISSING. ANY UNJUDGED cell (no comparator answered) declines the run with rc 2, because under the 17:19Z amendment "a missing cell is a NO-GO". ALL_WRONG (every engine wrong) is NAMED and counted per model (`summary.all_wrong_by_model`), never a violation and never a pass, per the cop''s ruling. The prompt set must declare a POSITIVE CONTROL (golden-2plus2, `"control": true`). A control cell that is ALL_WRONG, a model with no measured control cell, and a set that declares none each decline rc 2, with the cause named in `declined_because`. A comparator that exits non-zero or cannot be parsed is no oracle. A drift row pins the prompt set to golden_output.rs::golden_test_cases.' + - 'Mutation, 11 mutants, each turning the table RED: removing the RED rule breaks 6 rows; ignoring apr''s exit code breaks 2; ignoring fallback/backend breaks 1; skipping a missing apr row breaks 14; letting UNJUDGED pass breaks 4; `correct()` ignoring `answered` breaks 4; letting a control ALL_WRONG pass breaks 2; letting an undeclared control pass breaks 1; letting an unmeasured control pass breaks 1; not counting ALL_WRONG per model breaks 1; adding a pattern to the prompt set breaks 4, including the drift row. (An earlier no-GREEN decline was removed, not mutated: the per-model control rule subsumes it, and a mutant of it broke nothing.)' + - 'The first receipts are committed at evidence/crux/0.69.0/lambda-cpu.{json,md} and evidence/crux/0.69.0/lambda-gpu.{json,md}, measured by apr 0.69.0 and the harness at afb05a3de (`apr --version` and `harness.sha` in each receipt). MEASURED: lambda-cpu is PASS, 3 GREEN, with apr and llama.cpp both correct. lambda-gpu is RED, 2 RED + 1 GREEN: on golden-2plus2 (the control) and golden-paris, `apr run --gpu` exits 14 (CUDA diverges from CPU, cosine 0.8795 at position 3; wgpu rejected at 0.955; fell back to CPU and refused) while llama.cpp answers correctly on the same RTX 4090. golden-greeting ran on the GPU with no fallback and is GREEN. The RED cells are an instance of the open #3602 (same model, same host: the GPU result fails its own correctness check and falls back). They are named on #3739 and are not fixed in this PR. The harness''s job is to go RED here, and it did.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'Refs #3739, slice 1 (not a closing keyword; the issue has 6 done_when items and this meets parts of 1, 2 and 3). SCOPE AMENDMENT on the issue (issuecomment-5764587628, cop aprender-04, 17:19Z), verbatim: "Operator, verbatim: \"and CRUX for each verb in pre-release dogfood\". This applies the earlier operator line, \"the way to prevent testing leakage is tests are dervived from surface and SHACL enforced\" (#3745). The verb universe is DERIVED, not listed. It is every command in the surface the release-candidate binary emits (#3745 S1). The run/chat/serve/code list in this issue''s title is superseded as a scope; those four are the first slice, not the set. Every derived verb has a CRUX mapping entry in one committed correspondence file ... A derived verb with NO entry is a SHACL violation. Absence is never \"no comparator\". ... Pre-release, and it gates the tag. CRUX runs in the pre-publish dogfood (T-1), and its receipt is a `:CruxCell` input to `release-readiness-v1` (#3715/#3745). A cell where a comparator is right and apr is wrong is a NO-GO, and so is a missing cell." The cop''s ruling on sequencing, relayed the same minute: "The thin slice (qwen2.5-coder-1.5b `run`, lambda, 3 engines) stays first", and the correspondence file and `:CruxCell` fields are agreed with aprender-97 (#3745 S2, #3715) BEFORE they are coded. So this PR adds no correspondence file. The cop''s rulings of 18:47Z, verbatim: "(1) ALL_WRONG is NAMED, not a violation, since the bar is \"apr right wherever a competitor is right\". Two guards so it can''t become the escape: (i) it''s never scored Pass, and the receipt counts ALL_WRONG per model; (ii) every model''s prompt set includes a positive-control prompt that the smallest model answers right on every engine. If that control comes back ALL_WRONG, the run is DECLINED (rc 2) as a broken harness, never recorded as a model limitation." and "Every CRUX ollama request sets `keep_alive: 0`, runs under the GPU lock, and the cell checks VRAM is released before it drops the lock." Its ruling (2), the lambda ollama upgrade as a binary-only forjar change in paiml/infra, is slice 2 and is not in this PR. Its receipt already keys cells on the release-readiness tuple (model_sha256, host, verb, thinking, rung) plus prompt_id, as aprender-97 asked, and makes an UNJUDGED (unmeasured) cell decline. The issue''s done_when, verbatim: (1) "`scripts/crux_inference_dogfood.sh ` runs, per inventory Q4_K model × {lambda, gx10}, the SAME prompt set (the golden prompts + a consumer brief from #3716 at each declared context rung that fits) through: run: `apr run` · `llama-cli` · `ollama run`; chat (multi-turn): `apr chat` · llama-cli conversation · `ollama run` multi-turn; serve: `apr serve` · `llama-server` · `ollama serve`, all through the SAME OpenAI `/v1/chat/completions` client, streaming + non-streaming; code: `apr code` on a scripted edit-and-verify task, and the same task driven through llama-server and ollama by the same client; thinking ON and OFF where the model supports it (#3723), at temperature 0, same seed" (2) "per cell it records: answer correct (golden), exact-match agreement across engines, token-id parity (apr vs `llama-tokenize`), TTFT, decode tok/s, prompt/completion tokens, and the max context each engine accepts before refusing (or failing silently)" (3) "the gate: a cell where llama.cpp OR ollama answers correctly and apr doesn''t is RED (a finding naming the cell). Performance ratios are REPORTED, never claimed (the withdrawn-headline rule). The receipt feeds the SHACL shape (#3715) as a required cell kind." (4) "ollama is upgraded on lambda through the declared path (forjar) to a version that loads qwen35 and qwen3moe GGUFs; both hosts'' versions are recorded" (5) "runs in the release train (the T-1 `models` step, #3717) and ALSO on demand; the results table is committed per release (`evidence/crux//`)" (6) "depends on: #3563 (A), #3723, #3571". SLICING, ruled by the cop (aprender-04, 2026-09-21: "Start with a design note on #3739 and a first thin slice (qwen2.5-coder-1.5b Q4_K_M, `run` verb, lambda, all 3 engines) to prove the harness, then widen") and laid out in the design note: 1 run (this PR); 2 ollama on lambda via forjar + gx10; 3 chat; 4 serve through one client (perf via `apr test llm bench`, needs #3571); 5 code; 6 thinking ON (needs #3723); 7 consumer rungs + max context; 8 T-1 wiring (#3717). SCOPE: scripts/crux_inference_dogfood.sh (driver), scripts/lib/crux_inference_judge.py (judge), scripts/check_crux_inference_judge.sh (case table), scripts/crux_inference_prompts.json (prompt set), evidence/crux/0.69.0/*, this fragment. ONE DELIBERATE DEVIATION from the letter of done_when 2: token-id parity uses llama.cpp''s server `/tokenize` on its own `/apply-template` rendering, not the `llama-tokenize` binary. lambda''s pinned build dir (the one llama_bin.sh resolves) has no `llama-tokenize`, the server exists on both hosts, and both call the same llama_tokenize on the same vocab. Rendering first also makes parity cover the chat template, which is where #3672 lives. PERF-009: the driver starts engines but contains no rate token and reads no clock, and the judge starts nothing, so the competing-harness count stays 0 without an allowlist entry. The serve slice takes TTFT and decode rate from `apr test llm bench`, not from a new timer. FINDINGS on lambda, apr 0.69.0 @ 52f43da71, qwen2.5-coder-1.5b Q4_K_M sha256 cc324af0…: (a) apr renders the chat template twice (#3672): 46 prompt ids against llama.cpp''s 36, diverging at id 1, and the U+200B escapes encode as id 0 (#3726). The golden answer is still right ("4"), so only the parity field sees it. (b) ollama 0.5.7 imports a GGUF with no chat template; gx10''s 0.33.2 imports the GGUF''s own jinja template but stores blob b9a2f9f3…, not the input''s cc324af0…' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 3103603d5b..44fc6b714f 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -19967,3 +19967,27 @@ roadmap: labels: - kind:code notes: "Refs #3577 (ONT-4c3; its ledger binding is infra's row, bound after this merges) and #3624 (the by_entity_type sibling, not closed here). R-3 of ONT-001: pc_extract is one planted defect per registered extractor, every run." +- id: PMAT-3739 + github_issue: 3739 + item_type: task + title: 'CRUX inference dogfood slice 1: the SAME GGUF + SAME golden prompts through apr, llama.cpp@pin and ollama on the `run` verb, judged by one rule: a competitor right and apr wrong is RED' + status: in_progress + priority: critical + assigned_to: aprender-76 + created: 2026-09-21T17:10:00Z + updated: 2026-09-21T20:20:00Z + spec: null + acceptance_criteria: + - 'SLICE 1 of #3739, as the design note on the issue lays out (issuecomment-5764437831): the `run` verb, the golden prompts, thinking OFF, lambda. This PR does NOT meet the issue''s done_when in full, and it says `Refs #3739`, never a closing keyword. Every receipt it writes carries a `not_covered` list naming the verbs chat, serve and code, thinking ON, the consumer-brief rungs and max context, and TTFT, so a partial run cannot be read as a full one.' + - 'done_when 1, the part this slice meets, verbatim: "`scripts/crux_inference_dogfood.sh ` runs ... the SAME prompt set (the golden prompts ...) through: run: `apr run` · `llama-cli` · `ollama run`". Measured: `bash scripts/crux_inference_dogfood.sh 0.69.0 --backend cpu --model ~/models/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf` on lambda gives rc 0 with 3 cells GREEN, apr and llama.cpp@d1d3c3396 both correct. It declines (rc 2) when `apr --version` is not , when no --model is given, for a verb other than run, when --engines omits apr, and when no comparator resolves.' + - '"Same model" is recorded, not asserted, verbatim: "ollama IMPORTS the exact GGUF via a Modelfile (`FROM `) and the receipt records sha256 for all three engines". The receipt holds the input sha256 and ollama''s stored blob digest, with `blob_is_input`. An ollama import that carries no chat template (`TEMPLATE {{ .Prompt }}`, what 0.5.7 does on lambda) is REFUSED by name for every cell, never run as a raw completion against two engines that chat. GPU sharing, per the cop''s ruling: on the gpu lane one cell (one prompt through every engine) holds /tmp/apr-gpu.lock (fd 9, closed in every engine) for its engine invocations only. ollama runs with `--keepalive 0`, and the cell stops the model and waits for `ollama ps` to drop it before releasing the lock, recording `vram_released`. The cpu lane takes no lock (the rule''s clause 2); every engine runs under `choom -n 1000`.' + - 'done_when 2, the part this slice meets: per cell the receipt records answer correct (golden), exact-match agreement across engines, token-id parity (apr''s `-v` prompt ids against llama.cpp''s own `/apply-template` + `/tokenize` rendering of the same messages; count plus the id prefix apr prints, labelled `apr_ids_complete`), and prompt/completion tokens and rates as each engine REPORTED them, labelled by engine and never judged. The driver reads no clock and computes no rate (`check_no_competing_harnesses.sh` stays count=0). TTFT and max context belong to later slices and are listed as not covered.' + - 'done_when 3, the gate, verbatim: "a cell where llama.cpp OR ollama answers correctly and apr doesn''t is **RED** (a finding naming the cell)". `bash scripts/check_crux_inference_judge.sh`, which guard_tree runs bare, gives rc 0 and "21 ok, 0 broke". Its row 2 is the #3715 falsifier: llama.cpp right and apr wrong makes the judge exit 1 with the cell RED. Other rows make RED an apr that exits non-zero, falls back off the lane''s backend, or is MISSING. ANY UNJUDGED cell (no comparator answered) declines the run with rc 2, because under the 17:19Z amendment "a missing cell is a NO-GO". ALL_WRONG (every engine wrong) is NAMED and counted per model (`summary.all_wrong_by_model`), never a violation and never a pass, per the cop''s ruling. The prompt set must declare a POSITIVE CONTROL (golden-2plus2, `"control": true`). A control cell that is ALL_WRONG, a model with no measured control cell, and a set that declares none each decline rc 2, with the cause named in `declined_because`. A comparator that exits non-zero or cannot be parsed is no oracle. A drift row pins the prompt set to golden_output.rs::golden_test_cases.' + - 'Mutation, 11 mutants, each turning the table RED: removing the RED rule breaks 6 rows; ignoring apr''s exit code breaks 2; ignoring fallback/backend breaks 1; skipping a missing apr row breaks 14; letting UNJUDGED pass breaks 4; `correct()` ignoring `answered` breaks 4; letting a control ALL_WRONG pass breaks 2; letting an undeclared control pass breaks 1; letting an unmeasured control pass breaks 1; not counting ALL_WRONG per model breaks 1; adding a pattern to the prompt set breaks 4, including the drift row. (An earlier no-GREEN decline was removed, not mutated: the per-model control rule subsumes it, and a mutant of it broke nothing.)' + - 'The first receipts are committed at evidence/crux/0.69.0/lambda-cpu.{json,md} and evidence/crux/0.69.0/lambda-gpu.{json,md}, measured by apr 0.69.0 and the harness at afb05a3de (`apr --version` and `harness.sha` in each receipt). MEASURED: lambda-cpu is PASS, 3 GREEN, with apr and llama.cpp both correct. lambda-gpu is RED, 2 RED + 1 GREEN: on golden-2plus2 (the control) and golden-paris, `apr run --gpu` exits 14 (CUDA diverges from CPU, cosine 0.8795 at position 3; wgpu rejected at 0.955; fell back to CPU and refused) while llama.cpp answers correctly on the same RTX 4090. golden-greeting ran on the GPU with no fallback and is GREEN. The RED cells are an instance of the open #3602 (same model, same host: the GPU result fails its own correctness check and falls back). They are named on #3739 and are not fixed in this PR. The harness''s job is to go RED here, and it did.' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'Refs #3739, slice 1 (not a closing keyword; the issue has 6 done_when items and this meets parts of 1, 2 and 3). SCOPE AMENDMENT on the issue (issuecomment-5764587628, cop aprender-04, 17:19Z), verbatim: "Operator, verbatim: \"and CRUX for each verb in pre-release dogfood\". This applies the earlier operator line, \"the way to prevent testing leakage is tests are dervived from surface and SHACL enforced\" (#3745). The verb universe is DERIVED, not listed. It is every command in the surface the release-candidate binary emits (#3745 S1). The run/chat/serve/code list in this issue''s title is superseded as a scope; those four are the first slice, not the set. Every derived verb has a CRUX mapping entry in one committed correspondence file ... A derived verb with NO entry is a SHACL violation. Absence is never \"no comparator\". ... Pre-release, and it gates the tag. CRUX runs in the pre-publish dogfood (T-1), and its receipt is a `:CruxCell` input to `release-readiness-v1` (#3715/#3745). A cell where a comparator is right and apr is wrong is a NO-GO, and so is a missing cell." The cop''s ruling on sequencing, relayed the same minute: "The thin slice (qwen2.5-coder-1.5b `run`, lambda, 3 engines) stays first", and the correspondence file and `:CruxCell` fields are agreed with aprender-97 (#3745 S2, #3715) BEFORE they are coded. So this PR adds no correspondence file. The cop''s rulings of 18:47Z, verbatim: "(1) ALL_WRONG is NAMED, not a violation, since the bar is \"apr right wherever a competitor is right\". Two guards so it can''t become the escape: (i) it''s never scored Pass, and the receipt counts ALL_WRONG per model; (ii) every model''s prompt set includes a positive-control prompt that the smallest model answers right on every engine. If that control comes back ALL_WRONG, the run is DECLINED (rc 2) as a broken harness, never recorded as a model limitation." and "Every CRUX ollama request sets `keep_alive: 0`, runs under the GPU lock, and the cell checks VRAM is released before it drops the lock." Its ruling (2), the lambda ollama upgrade as a binary-only forjar change in paiml/infra, is slice 2 and is not in this PR. Its receipt already keys cells on the release-readiness tuple (model_sha256, host, verb, thinking, rung) plus prompt_id, as aprender-97 asked, and makes an UNJUDGED (unmeasured) cell decline. The issue''s done_when, verbatim: (1) "`scripts/crux_inference_dogfood.sh ` runs, per inventory Q4_K model × {lambda, gx10}, the SAME prompt set (the golden prompts + a consumer brief from #3716 at each declared context rung that fits) through: run: `apr run` · `llama-cli` · `ollama run`; chat (multi-turn): `apr chat` · llama-cli conversation · `ollama run` multi-turn; serve: `apr serve` · `llama-server` · `ollama serve`, all through the SAME OpenAI `/v1/chat/completions` client, streaming + non-streaming; code: `apr code` on a scripted edit-and-verify task, and the same task driven through llama-server and ollama by the same client; thinking ON and OFF where the model supports it (#3723), at temperature 0, same seed" (2) "per cell it records: answer correct (golden), exact-match agreement across engines, token-id parity (apr vs `llama-tokenize`), TTFT, decode tok/s, prompt/completion tokens, and the max context each engine accepts before refusing (or failing silently)" (3) "the gate: a cell where llama.cpp OR ollama answers correctly and apr doesn''t is RED (a finding naming the cell). Performance ratios are REPORTED, never claimed (the withdrawn-headline rule). The receipt feeds the SHACL shape (#3715) as a required cell kind." (4) "ollama is upgraded on lambda through the declared path (forjar) to a version that loads qwen35 and qwen3moe GGUFs; both hosts'' versions are recorded" (5) "runs in the release train (the T-1 `models` step, #3717) and ALSO on demand; the results table is committed per release (`evidence/crux//`)" (6) "depends on: #3563 (A), #3723, #3571". SLICING, ruled by the cop (aprender-04, 2026-09-21: "Start with a design note on #3739 and a first thin slice (qwen2.5-coder-1.5b Q4_K_M, `run` verb, lambda, all 3 engines) to prove the harness, then widen") and laid out in the design note: 1 run (this PR); 2 ollama on lambda via forjar + gx10; 3 chat; 4 serve through one client (perf via `apr test llm bench`, needs #3571); 5 code; 6 thinking ON (needs #3723); 7 consumer rungs + max context; 8 T-1 wiring (#3717). SCOPE: scripts/crux_inference_dogfood.sh (driver), scripts/lib/crux_inference_judge.py (judge), scripts/check_crux_inference_judge.sh (case table), scripts/crux_inference_prompts.json (prompt set), evidence/crux/0.69.0/*, this fragment. ONE DELIBERATE DEVIATION from the letter of done_when 2: token-id parity uses llama.cpp''s server `/tokenize` on its own `/apply-template` rendering, not the `llama-tokenize` binary. lambda''s pinned build dir (the one llama_bin.sh resolves) has no `llama-tokenize`, the server exists on both hosts, and both call the same llama_tokenize on the same vocab. Rendering first also makes parity cover the chat template, which is where #3672 lives. PERF-009: the driver starts engines but contains no rate token and reads no clock, and the judge starts nothing, so the competing-harness count stays 0 without an allowlist entry. The serve slice takes TTFT and decode rate from `apr test llm bench`, not from a new timer. FINDINGS on lambda, apr 0.69.0 @ 52f43da71, qwen2.5-coder-1.5b Q4_K_M sha256 cc324af0…: (a) apr renders the chat template twice (#3672): 46 prompt ids against llama.cpp''s 36, diverging at id 1, and the U+200B escapes encode as id 0 (#3726). The golden answer is still right ("4"), so only the parity field sees it. (b) ollama 0.5.7 imports a GGUF with no chat template; gx10''s 0.33.2 imports the GGUF''s own jinja template but stores blob b9a2f9f3…, not the input''s cc324af0…' diff --git a/evidence/crux/0.69.0/lambda-cpu.json b/evidence/crux/0.69.0/lambda-cpu.json new file mode 100644 index 0000000000..716ce9b2c5 --- /dev/null +++ b/evidence/crux/0.69.0/lambda-cpu.json @@ -0,0 +1,413 @@ +{ + "version": "0.69.0", + "host": "lambda", + "backend": "cpu", + "isa": "x86_64", + "gpu": "NVIDIA GeForce RTX 4090", + "engines": [ + "apr", + "llama.cpp", + "ollama" + ], + "verbs": [ + "run" + ], + "thinking": [ + "off" + ], + "sampling": { + "temperature": 0.0, + "seed": 42, + "context": 4096, + "max_tokens": 64, + "source": "scripts/llama_pin.toml [protocol]" + }, + "apr": { + "version_line": "apr 0.69.0 (afb05a3de)" + }, + "harness": { + "sha": "afb05a3de", + "driver": "scripts/crux_inference_dogfood.sh" + }, + "llama_cpp": { + "build": "version: 0.4.1-dev (build 10987, commit d1d3c3396)", + "pin": "d1d3c3396", + "unavailable": null + }, + "ollama": { + "server_version": "0.5.7", + "client_version": "0.5.7", + "unavailable": null + }, + "models": [ + { + "name": "qwen2.5-coder-1.5b-instruct-q4_k_m.gguf", + "sha256": "cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046", + "thinking_capable": false, + "ollama": { + "blob_sha256": "cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046", + "blob_is_input": true, + "template_sha256": "b507b9c2f6ca642bffcd06665ea7c91f235fd32daeefdf875a0f938db05fb315", + "refused": "ollama 0.5.7 imported the GGUF with no chat template (TEMPLATE {{ .Prompt }}), so it would complete the raw text rather than chat" + }, + "tokenization": { + "refused": null + } + } + ], + "not_covered": [ + "verbs chat, serve and code (the next slices of #3739)", + "thinking ON (apr has no toggle until #3723)", + "consumer-brief context rungs (#3716) and each engine's max accepted context", + "TTFT, which belongs to the serve verb's single OpenAI client" + ], + "schema": "crux-inference-receipt/v1", + "judged_at": "2026-09-21T17:41Z", + "cells": [ + { + "key": { + "model_sha256": "cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046", + "host": "lambda", + "verb": "run", + "thinking": "off", + "rung": "golden", + "prompt_id": "golden-2plus2" + }, + "verdict": "GREEN", + "expect_any": [ + "4" + ], + "engines": { + "apr": { + "answered": true, + "rc": 0, + "why": null, + "answer": "4", + "reported": { + "reported_by": "apr", + "prompt_tokens": 46, + "completion_tokens": 1, + "decode_rate_tokens_per_second": 0.4, + "inference_ms": 4961.73 + }, + "backend": { + "requested": "default", + "ran": "cpu", + "fell_back": false + }, + "prompt_ids": [ + 151644, + 872, + 198, + 27, + 0, + 0, + 0, + 96136, + 76, + 4906, + 91, + 29, + 872, + 198, + 3838, + 374, + 220, + 17, + 10, + 17, + 75414, + 0, + 0, + 0, + 96136, + 76, + 6213, + 91, + 397, + 27 + ], + "prompt_token_count": 46, + "rendered_prompt": "<|im_start|>user\\n<\\u{200b}|im_start|>user\\nWhat is 2+2?<\\u{200b}|im_end|>\\n<\\u{200b}|im_start|>assistant\\n<|im_end|>\\n<|im_start|>assistant\\n", + "correct": true + }, + "llama.cpp": { + "answered": true, + "rc": 0, + "why": null, + "answer": "2 + 2 equals 4.", + "reported": { + "reported_by": "llama.cpp", + "prompt_rate_tokens_per_second": 604.7, + "decode_rate_tokens_per_second": 71.8, + "prompt_tokens": null, + "completion_tokens": null + }, + "correct": true + }, + "ollama": { + "answered": false, + "rc": null, + "why": "refused: ollama 0.5.7 imported the GGUF with no chat template (TEMPLATE {{ .Prompt }}), so it would complete the raw text rather than chat", + "answer": null, + "reported": {}, + "correct": false + } + }, + "agreement": { + "answered": [ + "apr", + "llama.cpp" + ], + "all_identical": false, + "apr_matches": [] + }, + "token_parity": { + "measured": true, + "apr_count": 46, + "llama_cpp_count": 36, + "apr_ids_compared": 30, + "apr_ids_complete": false, + "first_divergence": 1, + "parity": false + } + }, + { + "key": { + "model_sha256": "cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046", + "host": "lambda", + "verb": "run", + "thinking": "off", + "rung": "golden", + "prompt_id": "golden-greeting" + }, + "verdict": "GREEN", + "expect_any": [ + "Hello", + "Hi", + "hey", + "hello", + "well", + "!" + ], + "engines": { + "apr": { + "answered": true, + "rc": 0, + "why": null, + "answer": "Hello! I'm just a computer program, so I don't have feelings, but I'm here to help you. How can I assist you today?", + "reported": { + "reported_by": "apr", + "prompt_tokens": 50, + "completion_tokens": 31, + "decode_rate_tokens_per_second": 6.5, + "inference_ms": 7154.17 + }, + "backend": { + "requested": "default", + "ran": "cpu", + "fell_back": false + }, + "prompt_ids": [ + 151644, + 872, + 198, + 27, + 0, + 0, + 0, + 96136, + 76, + 4906, + 91, + 29, + 872, + 198, + 9707, + 1052, + 11, + 1246, + 525, + 498, + 3730, + 3351, + 847, + 4238, + 75414, + 0, + 0, + 0, + 96136, + 76 + ], + "prompt_token_count": 50, + "rendered_prompt": "<|im_start|>user\\n<\\u{200b}|im_start|>user\\nHello there, how are you doing today my friend?<\\u{200b}|im_end|>\\n<\\u{200b}|im_start|>assistant\\n<|im_end|>\\n<|im_start|>assistant\\n", + "correct": true + }, + "llama.cpp": { + "answered": true, + "rc": 0, + "why": null, + "answer": "Hello! I'm doing well, thank you. How about yourself?", + "reported": { + "reported_by": "llama.cpp", + "prompt_rate_tokens_per_second": 534.7, + "decode_rate_tokens_per_second": 72.0, + "prompt_tokens": null, + "completion_tokens": null + }, + "correct": true + }, + "ollama": { + "answered": false, + "rc": null, + "why": "refused: ollama 0.5.7 imported the GGUF with no chat template (TEMPLATE {{ .Prompt }}), so it would complete the raw text rather than chat", + "answer": null, + "reported": {}, + "correct": false + } + }, + "agreement": { + "answered": [ + "apr", + "llama.cpp" + ], + "all_identical": false, + "apr_matches": [] + }, + "token_parity": { + "measured": true, + "apr_count": 50, + "llama_cpp_count": 40, + "apr_ids_compared": 30, + "apr_ids_complete": false, + "first_divergence": 1, + "parity": false + } + }, + { + "key": { + "model_sha256": "cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046", + "host": "lambda", + "verb": "run", + "thinking": "off", + "rung": "golden", + "prompt_id": "golden-paris" + }, + "verdict": "GREEN", + "expect_any": [ + "Paris" + ], + "engines": { + "apr": { + "answered": true, + "rc": 0, + "why": null, + "answer": "The capital of France is Paris.", + "reported": { + "reported_by": "apr", + "prompt_tokens": 46, + "completion_tokens": 7, + "decode_rate_tokens_per_second": 2.3, + "inference_ms": 5391.95 + }, + "backend": { + "requested": "default", + "ran": "cpu", + "fell_back": false + }, + "prompt_ids": [ + 151644, + 872, + 198, + 27, + 0, + 0, + 0, + 96136, + 76, + 4906, + 91, + 29, + 872, + 198, + 3838, + 374, + 279, + 6722, + 315, + 9625, + 75414, + 0, + 0, + 0, + 96136, + 76, + 6213, + 91, + 397, + 27 + ], + "prompt_token_count": 46, + "rendered_prompt": "<|im_start|>user\\n<\\u{200b}|im_start|>user\\nWhat is the capital of France?<\\u{200b}|im_end|>\\n<\\u{200b}|im_start|>assistant\\n<|im_end|>\\n<|im_start|>assistant\\n", + "correct": true + }, + "llama.cpp": { + "answered": true, + "rc": 0, + "why": null, + "answer": "The capital of France is Paris.", + "reported": { + "reported_by": "llama.cpp", + "prompt_rate_tokens_per_second": 616.3, + "decode_rate_tokens_per_second": 82.1, + "prompt_tokens": null, + "completion_tokens": null + }, + "correct": true + }, + "ollama": { + "answered": false, + "rc": null, + "why": "refused: ollama 0.5.7 imported the GGUF with no chat template (TEMPLATE {{ .Prompt }}), so it would complete the raw text rather than chat", + "answer": null, + "reported": {}, + "correct": false + } + }, + "agreement": { + "answered": [ + "apr", + "llama.cpp" + ], + "all_identical": true, + "apr_matches": [ + "llama.cpp" + ] + }, + "token_parity": { + "measured": true, + "apr_count": 46, + "llama_cpp_count": 36, + "apr_ids_compared": 30, + "apr_ids_complete": false, + "first_divergence": 1, + "parity": false + } + } + ], + "summary": { + "RED": 0, + "GREEN": 3, + "UNJUDGED": 0, + "ALL_WRONG": 0, + "cells": 3, + "judged": 3, + "verdict": "PASS", + "all_wrong_by_model": {}, + "controls": [ + "golden-2plus2" + ], + "declined_because": null + } +} diff --git a/evidence/crux/0.69.0/lambda-cpu.md b/evidence/crux/0.69.0/lambda-cpu.md new file mode 100644 index 0000000000..280df91c04 --- /dev/null +++ b/evidence/crux/0.69.0/lambda-cpu.md @@ -0,0 +1,16 @@ +# CRUX inference dogfood: 0.69.0 on lambda (cpu lane) + +apr `apr 0.69.0 (afb05a3de)` · llama.cpp `version: 0.4.1-dev (build 10987, commit d1d3c3396)` · ollama `0.5.7` · judged 2026-09-21T17:41Z + +**PASS**: 3 cells, 0 RED, 3 GREEN, 0 ALL_WRONG, 0 UNJUDGED. + +| model | verb | thinking | prompt | verdict | apr | llama.cpp | ollama | token parity | +|---|---|---|---|---|---|---|---|---| +| qwen2.5-coder-1.5b-instruct-q4_k_m.gguf | run | off | golden-2plus2 | **GREEN** | ✅ 4 | ✅ 2 + 2 equals 4. | ⛔ refused: ollama 0.5.7 imported the GGUF with no chat templa… | ≠ (apr 46 vs 36; first diff 1) | +| qwen2.5-coder-1.5b-instruct-q4_k_m.gguf | run | off | golden-greeting | **GREEN** | ✅ Hello! I'm just a computer program, so I don't … | ✅ Hello! I'm doing well, thank you. How about you… | ⛔ refused: ollama 0.5.7 imported the GGUF with no chat templa… | ≠ (apr 50 vs 40; first diff 1) | +| qwen2.5-coder-1.5b-instruct-q4_k_m.gguf | run | off | golden-paris | **GREEN** | ✅ The capital of France is Paris. | ✅ The capital of France is Paris. | ⛔ refused: ollama 0.5.7 imported the GGUF with no chat templa… | ≠ (apr 46 vs 36; first diff 1) | + +✅ correct · ❌ answered, wrong · ⛔ did not answer (reason shown). +Rates and token counts are in the JSON, as each engine reported them, and are not judged. + +Not covered by this run (the issue requires them): verbs chat, serve and code (the next slices of #3739); thinking ON (apr has no toggle until #3723); consumer-brief context rungs (#3716) and each engine's max accepted context; TTFT, which belongs to the serve verb's single OpenAI client. diff --git a/evidence/crux/0.69.0/lambda-gpu.json b/evidence/crux/0.69.0/lambda-gpu.json new file mode 100644 index 0000000000..8f2d3d527f --- /dev/null +++ b/evidence/crux/0.69.0/lambda-gpu.json @@ -0,0 +1,409 @@ +{ + "version": "0.69.0", + "host": "lambda", + "backend": "gpu", + "isa": "x86_64", + "gpu": "NVIDIA GeForce RTX 4090", + "engines": [ + "apr", + "llama.cpp", + "ollama" + ], + "verbs": [ + "run" + ], + "thinking": [ + "off" + ], + "sampling": { + "temperature": 0.0, + "seed": 42, + "context": 4096, + "max_tokens": 64, + "source": "scripts/llama_pin.toml [protocol]" + }, + "apr": { + "version_line": "apr 0.69.0 (afb05a3de)" + }, + "harness": { + "sha": "afb05a3de", + "driver": "scripts/crux_inference_dogfood.sh" + }, + "llama_cpp": { + "build": "version: 0.4.1-dev (build 10987, commit d1d3c3396)", + "pin": "d1d3c3396", + "unavailable": null + }, + "ollama": { + "server_version": "0.5.7", + "client_version": "0.5.7", + "unavailable": null + }, + "models": [ + { + "name": "qwen2.5-coder-1.5b-instruct-q4_k_m.gguf", + "sha256": "cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046", + "thinking_capable": false, + "ollama": { + "blob_sha256": "cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046", + "blob_is_input": true, + "template_sha256": "b507b9c2f6ca642bffcd06665ea7c91f235fd32daeefdf875a0f938db05fb315", + "refused": "ollama 0.5.7 imported the GGUF with no chat template (TEMPLATE {{ .Prompt }}), so it would complete the raw text rather than chat" + }, + "tokenization": { + "refused": null + } + } + ], + "not_covered": [ + "verbs chat, serve and code (the next slices of #3739)", + "thinking ON (apr has no toggle until #3723)", + "consumer-brief context rungs (#3716) and each engine's max accepted context", + "TTFT, which belongs to the serve verb's single OpenAI client" + ], + "schema": "crux-inference-receipt/v1", + "judged_at": "2026-09-21T18:15Z", + "cells": [ + { + "key": { + "model_sha256": "cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046", + "host": "lambda", + "verb": "run", + "thinking": "off", + "rung": "golden", + "prompt_id": "golden-2plus2" + }, + "verdict": "RED", + "expect_any": [ + "4" + ], + "engines": { + "apr": { + "answered": false, + "rc": 14, + "why": "exit 14", + "answer": "4", + "reported": { + "reported_by": "apr", + "prompt_tokens": 46, + "completion_tokens": 1, + "decode_rate_tokens_per_second": 0.1, + "inference_ms": 11280.96 + }, + "backend": { + "requested": "gpu", + "ran": "cpu", + "fell_back": true + }, + "prompt_ids": [ + 151644, + 872, + 198, + 27, + 0, + 0, + 0, + 96136, + 76, + 4906, + 91, + 29, + 872, + 198, + 3838, + 374, + 220, + 17, + 10, + 17, + 75414, + 0, + 0, + 0, + 96136, + 76, + 6213, + 91, + 397, + 27 + ], + "prompt_token_count": 46, + "rendered_prompt": "<|im_start|>user\\n<\\u{200b}|im_start|>user\\nWhat is 2+2?<\\u{200b}|im_end|>\\n<\\u{200b}|im_start|>assistant\\n<|im_end|>\\n<|im_start|>assistant\\n", + "correct": false + }, + "llama.cpp": { + "answered": true, + "rc": 0, + "why": null, + "answer": "2 + 2 equals 4.", + "reported": { + "reported_by": "llama.cpp", + "prompt_rate_tokens_per_second": 1455.4, + "decode_rate_tokens_per_second": 125.6, + "prompt_tokens": null, + "completion_tokens": null + }, + "correct": true + }, + "ollama": { + "answered": false, + "rc": null, + "why": "refused: ollama 0.5.7 imported the GGUF with no chat template (TEMPLATE {{ .Prompt }}), so it would complete the raw text rather than chat", + "answer": null, + "reported": {}, + "correct": false + } + }, + "agreement": { + "answered": [ + "llama.cpp" + ], + "all_identical": false, + "apr_matches": [] + }, + "token_parity": { + "measured": true, + "apr_count": 46, + "llama_cpp_count": 36, + "apr_ids_compared": 30, + "apr_ids_complete": false, + "first_divergence": 1, + "parity": false + } + }, + { + "key": { + "model_sha256": "cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046", + "host": "lambda", + "verb": "run", + "thinking": "off", + "rung": "golden", + "prompt_id": "golden-greeting" + }, + "verdict": "GREEN", + "expect_any": [ + "Hello", + "Hi", + "hey", + "hello", + "well", + "!" + ], + "engines": { + "apr": { + "answered": true, + "rc": 0, + "why": null, + "answer": "Hello! I'm just a computer program, so I don't have feelings, but I'm here to help you. How can I assist you today?", + "reported": { + "reported_by": "apr", + "prompt_tokens": 50, + "completion_tokens": 31, + "decode_rate_tokens_per_second": 9.2, + "inference_ms": 5655.37 + }, + "backend": { + "requested": "gpu", + "ran": "gpu", + "fell_back": false + }, + "prompt_ids": [ + 151644, + 872, + 198, + 27, + 0, + 0, + 0, + 96136, + 76, + 4906, + 91, + 29, + 872, + 198, + 9707, + 1052, + 11, + 1246, + 525, + 498, + 3730, + 3351, + 847, + 4238, + 75414, + 0, + 0, + 0, + 96136, + 76 + ], + "prompt_token_count": 50, + "rendered_prompt": "<|im_start|>user\\n<\\u{200b}|im_start|>user\\nHello there, how are you doing today my friend?<\\u{200b}|im_end|>\\n<\\u{200b}|im_start|>assistant\\n<|im_end|>\\n<|im_start|>assistant\\n", + "correct": true + }, + "llama.cpp": { + "answered": true, + "rc": 0, + "why": null, + "answer": "Hello! I'm doing well, thank you. How about yourself?", + "reported": { + "reported_by": "llama.cpp", + "prompt_rate_tokens_per_second": 1535.5, + "decode_rate_tokens_per_second": 193.4, + "prompt_tokens": null, + "completion_tokens": null + }, + "correct": true + }, + "ollama": { + "answered": false, + "rc": null, + "why": "refused: ollama 0.5.7 imported the GGUF with no chat template (TEMPLATE {{ .Prompt }}), so it would complete the raw text rather than chat", + "answer": null, + "reported": {}, + "correct": false + } + }, + "agreement": { + "answered": [ + "apr", + "llama.cpp" + ], + "all_identical": false, + "apr_matches": [] + }, + "token_parity": { + "measured": true, + "apr_count": 50, + "llama_cpp_count": 40, + "apr_ids_compared": 30, + "apr_ids_complete": false, + "first_divergence": 1, + "parity": false + } + }, + { + "key": { + "model_sha256": "cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046", + "host": "lambda", + "verb": "run", + "thinking": "off", + "rung": "golden", + "prompt_id": "golden-paris" + }, + "verdict": "RED", + "expect_any": [ + "Paris" + ], + "engines": { + "apr": { + "answered": false, + "rc": 14, + "why": "exit 14", + "answer": "The capital of France is Paris.", + "reported": { + "reported_by": "apr", + "prompt_tokens": 46, + "completion_tokens": 7, + "decode_rate_tokens_per_second": 0.8, + "inference_ms": 11645.83 + }, + "backend": { + "requested": "gpu", + "ran": "cpu", + "fell_back": true + }, + "prompt_ids": [ + 151644, + 872, + 198, + 27, + 0, + 0, + 0, + 96136, + 76, + 4906, + 91, + 29, + 872, + 198, + 3838, + 374, + 279, + 6722, + 315, + 9625, + 75414, + 0, + 0, + 0, + 96136, + 76, + 6213, + 91, + 397, + 27 + ], + "prompt_token_count": 46, + "rendered_prompt": "<|im_start|>user\\n<\\u{200b}|im_start|>user\\nWhat is the capital of France?<\\u{200b}|im_end|>\\n<\\u{200b}|im_start|>assistant\\n<|im_end|>\\n<|im_start|>assistant\\n", + "correct": false + }, + "llama.cpp": { + "answered": true, + "rc": 0, + "why": null, + "answer": "The capital of France is Paris.", + "reported": { + "reported_by": "llama.cpp", + "prompt_rate_tokens_per_second": 516.3, + "decode_rate_tokens_per_second": 306.5, + "prompt_tokens": null, + "completion_tokens": null + }, + "correct": true + }, + "ollama": { + "answered": false, + "rc": null, + "why": "refused: ollama 0.5.7 imported the GGUF with no chat template (TEMPLATE {{ .Prompt }}), so it would complete the raw text rather than chat", + "answer": null, + "reported": {}, + "correct": false + } + }, + "agreement": { + "answered": [ + "llama.cpp" + ], + "all_identical": false, + "apr_matches": [] + }, + "token_parity": { + "measured": true, + "apr_count": 46, + "llama_cpp_count": 36, + "apr_ids_compared": 30, + "apr_ids_complete": false, + "first_divergence": 1, + "parity": false + } + } + ], + "summary": { + "RED": 2, + "GREEN": 1, + "UNJUDGED": 0, + "ALL_WRONG": 0, + "cells": 3, + "judged": 3, + "verdict": "RED", + "all_wrong_by_model": {}, + "controls": [ + "golden-2plus2" + ], + "declined_because": null + } +} diff --git a/evidence/crux/0.69.0/lambda-gpu.md b/evidence/crux/0.69.0/lambda-gpu.md new file mode 100644 index 0000000000..9bb9f0e5db --- /dev/null +++ b/evidence/crux/0.69.0/lambda-gpu.md @@ -0,0 +1,16 @@ +# CRUX inference dogfood: 0.69.0 on lambda (gpu lane) + +apr `apr 0.69.0 (afb05a3de)` · llama.cpp `version: 0.4.1-dev (build 10987, commit d1d3c3396)` · ollama `0.5.7` · judged 2026-09-21T18:15Z + +**RED**: 3 cells, 2 RED, 1 GREEN, 0 ALL_WRONG, 0 UNJUDGED. + +| model | verb | thinking | prompt | verdict | apr | llama.cpp | ollama | token parity | +|---|---|---|---|---|---|---|---|---| +| qwen2.5-coder-1.5b-instruct-q4_k_m.gguf | run | off | golden-2plus2 | **RED** | ⛔ exit 14 | ✅ 2 + 2 equals 4. | ⛔ refused: ollama 0.5.7 imported the GGUF with no chat templa… | ≠ (apr 46 vs 36; first diff 1) | +| qwen2.5-coder-1.5b-instruct-q4_k_m.gguf | run | off | golden-greeting | **GREEN** | ✅ Hello! I'm just a computer program, so I don't … | ✅ Hello! I'm doing well, thank you. How about you… | ⛔ refused: ollama 0.5.7 imported the GGUF with no chat templa… | ≠ (apr 50 vs 40; first diff 1) | +| qwen2.5-coder-1.5b-instruct-q4_k_m.gguf | run | off | golden-paris | **RED** | ⛔ exit 14 | ✅ The capital of France is Paris. | ⛔ refused: ollama 0.5.7 imported the GGUF with no chat templa… | ≠ (apr 46 vs 36; first diff 1) | + +✅ correct · ❌ answered, wrong · ⛔ did not answer (reason shown). +Rates and token counts are in the JSON, as each engine reported them, and are not judged. + +Not covered by this run (the issue requires them): verbs chat, serve and code (the next slices of #3739); thinking ON (apr has no toggle until #3723); consumer-brief context rungs (#3716) and each engine's max accepted context; TTFT, which belongs to the serve verb's single OpenAI client. diff --git a/scripts/check_crux_inference_judge.sh b/scripts/check_crux_inference_judge.sh new file mode 100755 index 0000000000..816a74b8e2 --- /dev/null +++ b/scripts/check_crux_inference_judge.sh @@ -0,0 +1,310 @@ +#!/usr/bin/env bash +# check_crux_inference_judge.sh: the case table for the CRUX inference judge +# (scripts/lib/crux_inference_judge.py, #3739). Hermetic: every engine output +# is a fixture written under mktemp, so it needs no model, GPU or comparator +# and runs bare wherever guard_tree runs it. +# +# WHY A TABLE. The judge's one rule, "a comparator right and apr wrong is RED", +# is only a gate if it has been seen to go RED. Row 2 is the falsifier #3715 +# names: llama.cpp correct and apr wrong on one cell turns the run RED. The +# other rows pin each way apr can fail to answer (a wrong answer, a non-zero +# exit, a fallback off the lane's backend, a MISSING row), and each way a run +# can look fine while proving nothing: no comparator answered, the positive +# control came back ALL_WRONG or was never measured, no control was declared. +# ALL_WRONG anywhere else is named and counted per model, never a violation. +# +# It also refuses DRIFT between scripts/crux_inference_prompts.json and the +# golden cases it was derived from (golden_output.rs::golden_test_cases): the +# same user messages and expected patterns, in the same order. Zero cases found +# on either side is a refusal, never agreement. +# +# Exit: 0 every row behaved · 1 a row broke or the prompt set drifted · 2 ENV. +set -uo pipefail + +ROOT=$(cd "$(dirname "$0")/.." && pwd) || exit 2 +PROG=check_crux_inference_judge +command -v python3 >/dev/null 2>&1 || { printf '%s: ENV - python3 is missing\n' "$PROG" >&2; exit 2; } +JUDGE="$ROOT/scripts/lib/crux_inference_judge.py" +PROMPTS="$ROOT/scripts/crux_inference_prompts.json" +GOLDEN="$ROOT/crates/apr-cli/src/commands/golden_output.rs" +for f in "$JUDGE" "$PROMPTS" "$GOLDEN"; do + [ -f "$f" ] || { printf '%s: ENV - %s not found\n' "$PROG" "$f" >&2; exit 2; } +done + +TMP=$(mktemp -d) || exit 2 +_rm_tmp() { + case "${TMP:-}" in + /tmp/?*|/var/folders/?*) rm -rf -- "$TMP" || : ;; + *) : ;; + esac +} +trap _rm_tmp EXIT + +PASS=0 +FAIL=0 +ok() { printf ' ok %s\n' "$1"; PASS=$((PASS + 1)); } +broke(){ printf ' BROKE %s\n' "$1"; FAIL=$((FAIL + 1)); } + +# ---- fixture writers: one engine's raw output, in the shape it really prints --- +SHA=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa +apr_out() { # apr_out [ids] + printf '{"text": %s, "tokens_generated": 3, "tok_per_sec": 9.5, "backend": {"requested": "gpu", "ran": "%s", "fell_back": %s}}\n' \ + "$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$3")" "$4" "$5" > "$1/apr-$2.out" + printf '[DEBUG] formatted_prompt="<|im_start|>user\\nq<|im_end|>\\n"\n[DEBUG] add_bos=false, encoded %s tokens: [%s]\n' \ + "$(printf '%s' "${6:-1,2,3}" | tr ',' '\n' | grep -c .)" "${6:-1,2,3}" > "$1/apr-$2.err" +} +llama_out() { # llama_out — the pinned chat CLI's stdout + printf 'build : b10987-d1d3c3396\n\n> %s\n%s\n\n[ Prompt: 282.9 t/s | Generation: 104.9 t/s ]\n\nExiting...\n' "$3" "$4" > "$1/llama-$2.out" + : > "$1/llama-$2.err" +} +ollama_out() { # ollama_out + printf '%s\n' "$3" > "$1/ollama-$2.out" + printf 'total duration: 1.2s\nprompt eval count: 30 token(s)\neval count: 8 token(s)\neval rate: 90.00 tokens/s\n' > "$1/ollama-$2.err" +} +row() { # row [refused] + python3 - "$@" <<'PY' +import json, sys +m, eng, pid, rc, o, e = sys.argv[1:7] +ref = sys.argv[7] if len(sys.argv) > 7 else "" +open(m, "a").write(json.dumps({"kind": "gen", "engine": eng, "prompt_id": pid, "rc": int(rc) if rc else None, + "stdout": o or None, "stderr": e or None, "refused": ref or None, "model_sha256": "%s", + "host": "fixture", "verb": "run", "thinking": "off", "backend": "gpu"}) + "\n") +PY +} +tokrow() { # tokrow + printf '{"tokens": [%s]}\n' "$4" > "$2/ids-$3.json" + printf '{"kind": "tok", "engine": "llama.cpp", "model_sha256": "%%s", "prompt_id": "%s", "ids": "%s"}\n' "$3" "$2/ids-$3.json" >> "$1" +} +# The manifest rows carry the literal "%s" for the sha; fill it in one place. +seal() { sed -i "s/\"%s\"/\"$SHA\"/g" "$1"; } + +run_judge() { # run_judge — prints the receipt path; returns the judge's rc + printf '{"version": "0.0.0", "host": "fixture", "backend": "gpu", "engines": ["apr", "llama.cpp", "ollama"]}\n' > "$1/meta.json" + seal "$1/manifest.jsonl" + python3 "$JUDGE" collect --manifest "$1/manifest.jsonl" --prompts "$PROMPTS" --meta "$1/meta.json" \ + --out-json "$1/receipt.json" --out-md "$1/receipt.md" > "$1/judge.out" 2> "$1/judge.err" +} +verdict_of() { # verdict_of + python3 -c 'import json,sys +r = json.load(open(sys.argv[1])) +print(next((c["verdict"] for c in r["cells"] if c["key"]["prompt_id"] == sys.argv[2]), "ABSENT"))' "$1/receipt.json" "$2" 2>/dev/null +} +expect() { # expect + local rc=$3 got_v + got_v=$(verdict_of "$2" "$4") + if [ "$GOT_RC" = "$rc" ] && [ "$got_v" = "$5" ]; then ok "$1 (rc $GOT_RC, $4 $got_v)" + else broke "$1: want rc $rc + $5, got rc $GOT_RC + $got_v ($(tail -1 "$2/judge.err" 2>/dev/null))"; fi +} +newcase() { local d="$TMP/$1"; mkdir -p "$d"; : > "$d/manifest.jsonl"; printf '%s' "$d"; } + +Q="What is 2+2?" +P=golden-2plus2 + +printf '=== CRUX inference judge case table ===\n' + +# 1. every engine right: GREEN, and the run passes. +d=$(newcase all_right) +apr_out "$d" $P "2+2 equals 4." gpu false; llama_out "$d" $P "$Q" "2 + 2 equals 4."; ollama_out "$d" $P "4" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +row "$d/manifest.jsonl" ollama $P 0 "$d/ollama-$P.out" "$d/ollama-$P.err" +run_judge "$d"; GOT_RC=$? +expect "every engine right is GREEN and passes" "$d" 0 $P GREEN + +# 2. THE FALSIFIER: llama.cpp right, apr wrong. +d=$(newcase llama_right_apr_wrong) +apr_out "$d" $P "2+2 equals 5." gpu false; llama_out "$d" $P "$Q" "2 + 2 equals 4." +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +row "$d/manifest.jsonl" ollama $P "" "" "" "not requested here" +run_judge "$d"; GOT_RC=$? +expect "llama.cpp right and apr wrong is RED" "$d" 1 $P RED + +# 3. ollama right, apr exited 14 after falling back off the GPU. +d=$(newcase apr_fell_back) +apr_out "$d" $P "4" cpu true; ollama_out "$d" $P "4" +row "$d/manifest.jsonl" apr $P 14 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P "" "" "" "unresolved" +row "$d/manifest.jsonl" ollama $P 0 "$d/ollama-$P.out" "$d/ollama-$P.err" +run_judge "$d"; GOT_RC=$? +expect "apr right text but fell back and exited 14 is RED" "$d" 1 $P RED + +# 4. apr ran on another backend with rc 0 and fell_back=false: still not the lane. +d=$(newcase apr_wrong_backend) +apr_out "$d" $P "4" cpu false; llama_out "$d" $P "$Q" "4" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +run_judge "$d"; GOT_RC=$? +expect "apr on a backend the lane did not ask for is RED" "$d" 1 $P RED + +# 4b. apr printed a right-looking answer on the lane's backend and still exited non-zero. +d=$(newcase apr_nonzero_exit) +apr_out "$d" $P "4" gpu false; llama_out "$d" $P "$Q" "4" +row "$d/manifest.jsonl" apr $P 1 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +run_judge "$d"; GOT_RC=$? +expect "apr exiting non-zero is not an answer, however right it reads" "$d" 1 $P RED + +# 4c. the same on the comparator side: a comparator that exited non-zero cannot vouch. +d=$(newcase llama_nonzero_exit) +apr_out "$d" $P "5" gpu false; llama_out "$d" $P "$Q" "4" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 1 "$d/llama-$P.out" "$d/llama-$P.err" +run_judge "$d"; GOT_RC=$? +expect "a comparator exiting non-zero is no oracle (UNJUDGED)" "$d" 2 $P UNJUDGED + +# 5. the apr row is MISSING while llama.cpp answered right: absence is a violation. +d=$(newcase apr_missing) +llama_out "$d" $P "$Q" "4" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +run_judge "$d"; GOT_RC=$? +expect "a missing apr row where llama.cpp is right is RED" "$d" 1 $P RED + +# 6. no comparator answered: UNJUDGED, and a run of only that declines. +d=$(newcase no_oracle) +apr_out "$d" $P "4" gpu false +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P "" "" "" "unresolved: no_binary_named" +row "$d/manifest.jsonl" ollama $P "" "" "" "imported with no chat template" +run_judge "$d"; GOT_RC=$? +expect "no comparator answered is UNJUDGED and declines" "$d" 2 $P UNJUDGED + +# 7. everyone answered wrong on the control prompt: ALL_WRONG, and the run declines. +d=$(newcase all_wrong) +apr_out "$d" $P "5" gpu false; llama_out "$d" $P "$Q" "22"; ollama_out "$d" $P "five" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +row "$d/manifest.jsonl" ollama $P 0 "$d/ollama-$P.out" "$d/ollama-$P.err" +run_judge "$d"; GOT_RC=$? +expect "everyone wrong is ALL_WRONG and declines" "$d" 2 $P ALL_WRONG + +# 8. a comparator whose output cannot be parsed did not answer; it cannot vouch. +d=$(newcase llama_unparsed) +apr_out "$d" $P "5" gpu false +printf 'banner only, the turn never finished\n' > "$d/llama-$P.out"; : > "$d/llama-$P.err" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +run_judge "$d"; GOT_RC=$? +expect "an unparseable comparator is no oracle (UNJUDGED)" "$d" 2 $P UNJUDGED + +# 9. one RED among GREENs still fails the whole run. +d=$(newcase one_red_among_green) +for p in golden-2plus2 golden-paris; do + case $p in golden-2plus2) q="$Q"; a_apr="4" ;; golden-paris) q="What is the capital of France?"; a_apr="Lyon" ;; esac + case $p in golden-2plus2) a_ref="4" ;; golden-paris) a_ref="Paris" ;; esac + apr_out "$d" $p "$a_apr" gpu false; llama_out "$d" $p "$q" "$a_ref" + row "$d/manifest.jsonl" apr $p 0 "$d/apr-$p.out" "$d/apr-$p.err" + row "$d/manifest.jsonl" llama.cpp $p 0 "$d/llama-$p.out" "$d/llama-$p.err" +done +run_judge "$d"; GOT_RC=$? +expect "one RED cell among GREEN ones fails the run" "$d" 1 golden-paris RED + +# 9b. one UNJUDGED cell among GREEN ones: that cell was never compared, and the +# amended scope makes a missing cell a NO-GO, so the run declines. +d=$(newcase one_unjudged_among_green) +apr_out "$d" golden-2plus2 "4" gpu false; llama_out "$d" golden-2plus2 "$Q" "4" +row "$d/manifest.jsonl" apr golden-2plus2 0 "$d/apr-golden-2plus2.out" "$d/apr-golden-2plus2.err" +row "$d/manifest.jsonl" llama.cpp golden-2plus2 0 "$d/llama-golden-2plus2.out" "$d/llama-golden-2plus2.err" +apr_out "$d" golden-paris "Paris" gpu false +row "$d/manifest.jsonl" apr golden-paris 0 "$d/apr-golden-paris.out" "$d/apr-golden-paris.err" +row "$d/manifest.jsonl" llama.cpp golden-paris 124 "" "" "" +run_judge "$d"; GOT_RC=$? +expect "one UNJUDGED cell among GREEN ones declines the run" "$d" 2 golden-paris UNJUDGED + +# 9c-9e. ALL_WRONG is NAMED, not a violation (the cop's ruling), and the positive +# control keeps it from becoming the escape. +# 9c. a non-control ALL_WRONG beside a GREEN control passes, and is counted per model. +d=$(newcase all_wrong_named) +apr_out "$d" golden-2plus2 "4" gpu false; llama_out "$d" golden-2plus2 "$Q" "4" +row "$d/manifest.jsonl" apr golden-2plus2 0 "$d/apr-golden-2plus2.out" "$d/apr-golden-2plus2.err" +row "$d/manifest.jsonl" llama.cpp golden-2plus2 0 "$d/llama-golden-2plus2.out" "$d/llama-golden-2plus2.err" +apr_out "$d" golden-paris "Lyon" gpu false; llama_out "$d" golden-paris "What is the capital of France?" "Marseille" +row "$d/manifest.jsonl" apr golden-paris 0 "$d/apr-golden-paris.out" "$d/apr-golden-paris.err" +row "$d/manifest.jsonl" llama.cpp golden-paris 0 "$d/llama-golden-paris.out" "$d/llama-golden-paris.err" +run_judge "$d"; GOT_RC=$? +expect "a non-control ALL_WRONG is named and the run still passes" "$d" 0 golden-paris ALL_WRONG +got=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["summary"]["all_wrong_by_model"])' "$d/receipt.json" 2>/dev/null) +[ "$got" = "{'$SHA': 1}" ] && ok "ALL_WRONG is counted per model (1)" || broke "all_wrong_by_model: got '$got'" + +# 9d. the positive control comes back ALL_WRONG: a broken harness, so the run declines. +d=$(newcase control_all_wrong) +apr_out "$d" golden-2plus2 "22" gpu false; llama_out "$d" golden-2plus2 "$Q" "22" +row "$d/manifest.jsonl" apr golden-2plus2 0 "$d/apr-golden-2plus2.out" "$d/apr-golden-2plus2.err" +row "$d/manifest.jsonl" llama.cpp golden-2plus2 0 "$d/llama-golden-2plus2.out" "$d/llama-golden-2plus2.err" +apr_out "$d" golden-paris "Paris" gpu false; llama_out "$d" golden-paris "What is the capital of France?" "Paris" +row "$d/manifest.jsonl" apr golden-paris 0 "$d/apr-golden-paris.out" "$d/apr-golden-paris.err" +row "$d/manifest.jsonl" llama.cpp golden-paris 0 "$d/llama-golden-paris.out" "$d/llama-golden-paris.err" +run_judge "$d"; GOT_RC=$? +expect "the positive control ALL_WRONG declines the run beside a GREEN cell" "$d" 2 golden-2plus2 ALL_WRONG + +# 9e. a prompt set that declares no positive control declines, however green the cells. +d=$(newcase no_control_declared) +python3 -c 'import json,sys +d = json.load(open(sys.argv[1])) +for p in d["prompts"]: p.pop("control", None) +json.dump(d, open(sys.argv[2], "w"))' "$PROMPTS" "$d/prompts.json" +apr_out "$d" $P "4" gpu false; llama_out "$d" $P "$Q" "4" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +PROMPTS_SAVED=$PROMPTS; PROMPTS="$d/prompts.json" +run_judge "$d"; GOT_RC=$? +PROMPTS=$PROMPTS_SAVED +expect "a prompt set with no positive control declines" "$d" 2 $P GREEN +why=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["summary"]["declined_because"])' "$d/receipt.json" 2>/dev/null) +case "$why" in + *"declares no positive control"*) ok "the decline names the undeclared control, not a symptom of it" ;; + *) broke "undeclared control: declined_because was '$why'" ;; +esac + +# 9f. a model whose run never measured the control prompt declines, however green: +# an unmeasured control proves nothing about the harness. +d=$(newcase no_control_cell) +apr_out "$d" golden-paris "Paris" gpu false; llama_out "$d" golden-paris "What is the capital of France?" "Paris" +row "$d/manifest.jsonl" apr golden-paris 0 "$d/apr-golden-paris.out" "$d/apr-golden-paris.err" +row "$d/manifest.jsonl" llama.cpp golden-paris 0 "$d/llama-golden-paris.out" "$d/llama-golden-paris.err" +run_judge "$d"; GOT_RC=$? +expect "a model with no measured control cell declines" "$d" 2 golden-paris GREEN + +# 10-11. token parity: equal ids agree; a divergence is located, never averaged away. +d=$(newcase parity) +apr_out "$d" $P "4" gpu false "151644,872,198,3838"; llama_out "$d" $P "$Q" "4" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +tokrow "$d/manifest.jsonl" "$d" $P "151644,872,198,3838" +run_judge "$d"; GOT_RC=$? +got=$(python3 -c 'import json,sys; t=json.load(open(sys.argv[1]))["cells"][0]["token_parity"]; print(t.get("parity"), t.get("first_divergence"))' "$d/receipt.json" 2>/dev/null) +[ "$got" = "True None" ] && ok "identical prompt ids are parity" || broke "identical prompt ids: got '$got'" +d=$(newcase parity_diverges) +apr_out "$d" $P "4" gpu false "151644,872,198,27,0,0,0"; llama_out "$d" $P "$Q" "4" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +tokrow "$d/manifest.jsonl" "$d" $P "151644,872,198,3838,374" +run_judge "$d"; GOT_RC=$? +got=$(python3 -c 'import json,sys; t=json.load(open(sys.argv[1]))["cells"][0]["token_parity"]; print(t.get("parity"), t.get("first_divergence"))' "$d/receipt.json" 2>/dev/null) +[ "$got" = "False 3" ] && ok "a divergent prompt is located at its first differing id (3)" || broke "divergent prompt ids: got '$got'" + +# 12. the prompt set has not drifted from the golden cases it was derived from. +drift=$(python3 - "$GOLDEN" "$PROMPTS" <<'PY' +import json, re, sys +src = open(sys.argv[1]).read() +m = re.search(r"fn golden_test_cases\(\)[^{]*\{(.*?)\n\}", src, re.S) +if not m: + print("golden_test_cases() not found in %s" % sys.argv[1]); sys.exit(0) +body = re.sub(r"//[^\n]*", "", m.group(1)) +lit = r'"((?:[^"\\]|\\.)*)"' +cases = re.findall(r'\(\s*' + lit + r'\s*,\s*vec!\[(.*?)\]\s*,?\s*\)', body, re.S) +golden = [] +for prompt, pats in cases: + u = re.fullmatch(r"<\|im_start\|>user\\n(.*)<\|im_end\|>\\n<\|im_start\|>assistant\\n", prompt, re.S) + golden.append((u.group(1) if u else None, re.findall(lit, pats))) +mine = [(p["messages"][-1]["content"], p["expect_any"]) for p in json.load(open(sys.argv[2]))["prompts"] if p["rung"] == "golden"] +if not golden or not mine: + print("refused: %d golden case(s), %d prompt(s); zero on either side is not agreement" % (len(golden), len(mine))) +elif golden != mine: + print("DRIFT golden=%r prompts=%r" % (golden, mine)) +PY +) +[ -z "$drift" ] && ok "the prompt set matches golden_test_cases (3 cases, same order)" || broke "prompt set vs golden: $drift" + +printf '%s: %d ok, %d broke\n' "$PROG" "$PASS" "$FAIL" +[ "$FAIL" -eq 0 ] diff --git a/scripts/crux_inference_dogfood.sh b/scripts/crux_inference_dogfood.sh new file mode 100755 index 0000000000..24f60f0d76 --- /dev/null +++ b/scripts/crux_inference_dogfood.sh @@ -0,0 +1,460 @@ +#!/usr/bin/env bash +# crux_inference_dogfood.sh: the CRUX inference dogfood (#3739). The SAME GGUF +# and the SAME prompts go through apr, llama.cpp at the pin and ollama, and one +# rule judges them: a cell a comparator answers correctly and apr does not is RED. +# +# WHY. Every apr gate compared apr to apr, and so missed a tokenizer that never +# did real BPE (#3726) and a chat template applied twice (#3672): both sides of +# every comparison shared the defect. A competitor on the same file is the +# external oracle. Operator, verbatim: "same prompts/verbs on llama.cpp and say +# ollama for same model: chat/serve/run/code, etc". +# +# Usage: bash scripts/crux_inference_dogfood.sh --model [--model ]... +# [--host ] [--backend gpu|cpu] [--engines apr,llama.cpp,ollama] +# [--verbs run] [--out ] [--prompts ] [--timeout ] +# [--keep-ollama-models] [--keep-work] +# the apr version under test; `apr --version` must report it, or +# the run declines (a dogfood that measured another version is #3708) +# --backend the lane, applied to ALL three engines (default gpu): apr --gpu, +# llama.cpp -ngl 999, ollama's default; cpu = --no-gpu, -ngl 0, num_gpu 0. +# An apr that falls back from the lane's backend did not answer the cell. +# --out receipt dir (default evidence/crux/); writes -.{json,md} +# +# Exit: 0 no RED, no UNJUDGED cell, every model's positive control measured and +# not ALL_WRONG · 1 any RED · 2 decline (a cell no comparator answered, a broken +# or missing control, apr unpinned or the wrong version, a verb this slice +# cannot drive). ALL_WRONG elsewhere is named and counted per model. +# +# WHAT "SAME" MEANS, and how each engine is resolved: +# model one file by path. ollama IMPORTS that file through a Modelfile +# (`FROM `); a registry pull is a different file. The receipt +# records the input's sha256 and the blob digest ollama stored, which +# newer ollama versions do not keep byte-identical to the input. +# prompts scripts/crux_inference_prompts.json, as chat MESSAGES. Every engine +# applies the template it reads from the GGUF; that is under test too. +# apr scripts/apr_bin.sh (HEAD-built), or DOGFOOD_ALLOW_UNPINNED=1 plus $APR +# for a published binary, as model_ladder.sh does. +# llama scripts/llama_bin.sh ONLY ($LLAMA_BENCH_PATH names the pinned build). +# ollama $OLLAMA_BIN, else the forjar-declared $HOME/.local/bin/ollama, else +# /usr/local/bin/ollama; the SERVER's /api/version is what is recorded. +# sampling temperature, seed and context come from scripts/llama_pin.toml +# [protocol], the declaration the parity gates already read. +# +# NO RATE IS COMPUTED HERE. The judge (scripts/lib/crux_inference_judge.py) +# copies the token counts and rates each engine prints about itself into the +# receipt, labelled by engine, and judges none of them. Nothing in this file +# reads a clock. A throughput CLAIM goes through scripts/perf_gate.sh (PERF-009). +# +# GPU sharing (the cop's rule, /mnt/nvme-raid0/agent-wt/gpu-lock-rule.txt): +# on the gpu lane each cell holds /tmp/apr-gpu.lock for its engine invocations +# only, and every engine runs under `choom -n 1000`, so a measurement is the OOM +# victim, never a CI runner. llama.cpp and ollama are held to it as well as apr, +# and ollama runs with keep_alive 0 and must leave VRAM before the lock drops. +# +# Every status is captured as `cmd; rc=$?`, never through a pipe (#2336, #2360). +set -uo pipefail + +ROOT=$(cd "$(dirname "$0")/.." && pwd) || exit 2 +cd "$ROOT" || exit 2 + +decline() { printf 'decline: %s\n' "$*" >&2; exit 2; } + +VERSION="" +HOST_ID="" +BACKEND="gpu" +ENGINES="apr,llama.cpp,ollama" +VERBS="run" +OUT_DIR="" +PROMPTS="scripts/crux_inference_prompts.json" +TMO=600 +LOCK_WAIT=3600 +KEEP_OLLAMA=0 +KEEP_WORK=0 +MODELS=() +while [ $# -gt 0 ]; do + case "$1" in + --model) [ $# -ge 2 ] || decline "--model needs a value"; MODELS+=("$2"); shift 2 ;; + --host) [ $# -ge 2 ] || decline "--host needs a value"; HOST_ID="$2"; shift 2 ;; + --backend) [ $# -ge 2 ] || decline "--backend needs a value"; BACKEND="$2"; shift 2 ;; + --engines) [ $# -ge 2 ] || decline "--engines needs a value"; ENGINES="$2"; shift 2 ;; + --verbs) [ $# -ge 2 ] || decline "--verbs needs a value"; VERBS="$2"; shift 2 ;; + --out) [ $# -ge 2 ] || decline "--out needs a value"; OUT_DIR="$2"; shift 2 ;; + --prompts) [ $# -ge 2 ] || decline "--prompts needs a value"; PROMPTS="$2"; shift 2 ;; + --timeout) [ $# -ge 2 ] || decline "--timeout needs a value"; TMO="$2"; shift 2 ;; + --keep-ollama-models) KEEP_OLLAMA=1; shift ;; + --keep-work) KEEP_WORK=1; shift ;; + -h|--help) sed -n '2,24p' "$0"; exit 0 ;; + -*) decline "unknown argument '$1'" ;; + *) [ -z "$VERSION" ] || decline "one version, got '$VERSION' and '$1'"; VERSION="$1"; shift ;; + esac +done +[ -n "$VERSION" ] || decline "usage: $0 --model ..." +[ "${#MODELS[@]}" -gt 0 ] || decline "no --model given; this slice takes models by path" +case "$BACKEND" in gpu|cpu) ;; *) decline "--backend is gpu or cpu, got '$BACKEND'" ;; esac +case ",$VERBS," in + ,run,) ;; + *) decline "verbs '$VERBS': this slice drives 'run' only; chat/serve/code are the next slices of #3739 and are listed as not covered in every receipt" ;; +esac +want() { case ",$ENGINES," in *",$1,"*) return 0 ;; *) return 1 ;; esac; } +want apr || decline "apr is the subject; --engines must include it" +[ -f "$PROMPTS" ] || decline "prompt set $PROMPTS not found" + +# ---- apr: pinned, and the version asked for ------------------------------------ +if [ "${DOGFOOD_ALLOW_UNPINNED:-0}" = "1" ] && [ -n "${APR:-}" ]; then + : # published-binary mode: the caller pinned $APR deliberately +else + # shellcheck disable=SC1091 + . scripts/apr_bin.sh || decline "scripts/apr_bin.sh could not pin a HEAD-built apr" +fi +[ -x "${APR:-}" ] || decline "\$APR is not executable: '${APR:-}'" +APR_VERSION_LINE=$("$APR" --version 2>/dev/null | head -1) +APR_GOT=$(printf '%s\n' "$APR_VERSION_LINE" | sed -n 's/^apr \([0-9][0-9A-Za-z.+-]*\).*/\1/p') +[ "$APR_GOT" = "$VERSION" ] || decline "apr under test reports '$APR_VERSION_LINE'; this dogfood was asked for $VERSION" + +host_id() { + if [ -n "$HOST_ID" ]; then printf '%s' "$HOST_ID"; return; fi + local h; h=$(hostname 2>/dev/null || echo unknown) + case "$h" in + gx10*) printf 'gx10' ;; + *[Ll]ambda*) printf 'lambda' ;; + *) printf '%s' "$h" ;; + esac +} +HOST=$(host_id) +[ -n "$OUT_DIR" ] || OUT_DIR="evidence/crux/$VERSION" + +# ---- llama.cpp: the pin, through the one resolver --------------------------------- +# shellcheck disable=SC1091 +. scripts/llama_bin.sh >/dev/null 2>&1 +LLAMA_RC=$? +TEMP=$(llama_pin_get_raw temperature 2>/dev/null) +SEED=$(llama_pin_get_raw seed 2>/dev/null) +CTX=$(llama_pin_get_raw context_length 2>/dev/null) +[ -n "$TEMP" ] && [ -n "$SEED" ] && [ -n "$CTX" ] || decline "scripts/llama_pin.toml [protocol] temperature/seed/context_length unreadable" +LLAMA_OK=0 +LLAMA_WHY="" +if want llama.cpp; then + if [ "$LLAMA_RC" -ne 0 ]; then + LLAMA_WHY="llama.cpp unresolved: ${LLAMA_PIN_REASON:-unknown} (llama_bin rc $LLAMA_RC)" + elif [ -z "${LLAMA_CLI:-}" ] || [ -z "${LLAMA_SERVER:-}" ]; then + LLAMA_WHY="the pinned build has no chat CLI or no server binary beside it" + else + LLAMA_OK=1 + fi +fi + +# ---- ollama: resolved by declaration, recorded by what the server says ---------- +OLLAMA="" +OLLAMA_OK=0 +OLLAMA_WHY="" +OLLAMA_CLIENT="" +OLLAMA_SERVER="" +OLLAMA_HOST_URL="http://${OLLAMA_HOST:-127.0.0.1:11434}" +if want ollama; then + for cand in "${OLLAMA_BIN:-}" "$HOME/.local/bin/ollama" /usr/local/bin/ollama; do + [ -n "$cand" ] && [ -f "$cand" ] && [ -x "$cand" ] && { OLLAMA="$cand"; break; } + done + if [ -z "$OLLAMA" ]; then + OLLAMA_WHY="no ollama binary at \$OLLAMA_BIN, ~/.local/bin or /usr/local/bin" + else + OLLAMA_CLIENT=$("$OLLAMA" --version 2>&1 | sed -n 's/.*client version is \([0-9.]*\).*/\1/p;s/^ollama version is \([0-9.]*\).*/\1/p' | head -1) + OLLAMA_SERVER=$(curl -sf "$OLLAMA_HOST_URL/api/version" 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin).get("version",""))' 2>/dev/null) + if [ -z "$OLLAMA_SERVER" ]; then + OLLAMA_WHY="the ollama server at $OLLAMA_HOST_URL did not answer /api/version" + else + OLLAMA_OK=1 + fi + fi +fi +[ "$LLAMA_OK" = 1 ] || [ "$OLLAMA_OK" = 1 ] || decline "no comparator: llama.cpp (${LLAMA_WHY:-not requested}); ollama (${OLLAMA_WHY:-not requested})" + +# ---- work dir ------------------------------------------------------------------- +WORK=$(mktemp -d) || decline "mktemp failed" +SRV_PID="" +# The delete is guarded (SEC011): only a path under a temp root is removed. +OL_NAME="" +_cleanup() { + [ -n "$SRV_PID" ] && kill "$SRV_PID" 2>/dev/null + # A run killed mid-model leaves its ollama import behind unless removed here. + if [ -n "$OL_NAME" ] && [ "$OLLAMA_OK" = 1 ] && [ "$KEEP_OLLAMA" = 0 ]; then + "$OLLAMA" rm "$OL_NAME" > /dev/null 2>&1 + fi + [ "$KEEP_WORK" = 1 ] && { printf 'work kept: %s\n' "$WORK" >&2; return 0; } + local v="${WORK:-}" + case "$v" in + /tmp/?*|/var/folders/?*) if [ -n "$v" ] && [ "$v" != "/" ]; then rm -rf -- "$v" || :; fi ;; + *) return 0 ;; + esac +} +trap _cleanup EXIT +MANIFEST="$WORK/manifest.jsonl"; : > "$MANIFEST" +MODELS_JSONL="$WORK/models.jsonl"; : > "$MODELS_JSONL" + +# One file per prompt: its id list, and each prompt's last user message. +PIDS=$(python3 - "$PROMPTS" "$WORK" <<'PY' +import json, sys +d = json.load(open(sys.argv[1])) +for p in d["prompts"]: + open("%s/prompt-%s.txt" % (sys.argv[2], p["id"]), "w").write(p["messages"][-1]["content"]) + json.dump({"messages": p["messages"]}, open("%s/messages-%s.json" % (sys.argv[2], p["id"]), "w")) + print(p["id"]) +PY +) || decline "prompt set $PROMPTS unreadable" +MAXTOK=$(python3 -c 'import json,sys; print(int(json.load(open(sys.argv[1]))["max_tokens"]))' "$PROMPTS") || decline "max_tokens unreadable" + +# THE CELL LOCK (the cop's GPU rule rev 2, plus its #3739 ruling on ollama). On +# the gpu lane one cell, meaning one prompt through every engine, holds +# /tmp/apr-gpu.lock for exactly its engine invocations. It releases the lock only +# after ollama's model has left VRAM: ollama keeps a model resident (keep_alive, +# default 5 min), and a resident model would starve the next session's cell on a +# 24 GB card. flock exits 75 (not 1) when the lock is not had in time, so a lock +# wait is never read as an engine's own failure. The cpu lane touches no GPU +# compute and takes no lock (the rule's clause 2); every engine still runs under +# choom, so a measurement is the OOM victim, never a CI runner. +cell_lock() { + [ "$BACKEND" = gpu ] || return 0 + exec 9> /tmp/apr-gpu.lock || return 75 + flock -w "$LOCK_WAIT" -E 75 9 +} +cell_unlock() { + [ "$BACKEND" = gpu ] || return 0 + flock -u 9 + exec 9>&- +} + +# eng_run cmd... — one engine invocation. `9>&-` keeps the lock +# fd out of the engine, so nothing an engine leaves running can hold the lock. +eng_run() { + local o="$1" e="$2"; shift 2 + choom -n 1000 -- timeout "$TMO" "$@" > "$o" 2> "$e" < /dev/null 9>&- +} + +# ollama_unloaded : stop the model, then wait up to 30 s for `ollama ps` to +# stop listing it. Prints true or false, and the cell records which. +ollama_unloaded() { + local i listed + "$OLLAMA" stop "$1" > /dev/null 2>&1 9>&- + for i in $(seq 1 30); do + listed=$("$OLLAMA" ps 2>/dev/null 9>&-) + case "$listed" in *"$1"*) sleep 1 ;; *) printf 'true'; return 0 ;; esac + done + printf 'false' +} + +emit_gen() { # emit_gen [ollama_unloaded] + python3 - "$MANIFEST" "$1" "$2" "$3" "$4" "$5" "$6" "${7:-}" "$SHA" "$HOST" "$VERB" "$THINK" "$BACKEND" 9>&- <<'PY' +import json, sys +m, eng, pid, rc, o, e, ref, unl, sha, host, verb, think, be = sys.argv[1:14] +row = {"kind": "gen", "engine": eng, "prompt_id": pid, + "rc": int(rc) if rc.lstrip("-").isdigit() else None, + "stdout": o or None, "stderr": e or None, "refused": ref or None, + "model_sha256": sha, "host": host, "verb": verb, "thinking": think, "backend": be} +if unl: + row["ollama_unloaded"] = unl == "true" +open(m, "a").write(json.dumps(row) + "\n") +PY +} + +free_port() { python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])'; } + +# llama.cpp's OWN rendering and tokenization of each prompt: the reference apr's +# prompt ids are compared against. The server runs on the CPU (-ngl 0), only +# long enough to answer /props, /apply-template and /tokenize. +llama_tokenize_prompts() { # sets THINKING_CAPABLE; writes tok rows + local port d="$WORK/$SHA12/tok" pid ok=0 i + mkdir -p "$d" + port=$(free_port) || return 1 + choom -n 1000 -- "$LLAMA_SERVER" -m "$M" -ngl 0 -c 512 --no-warmup --host 127.0.0.1 --port "$port" \ + > "$d/server.log" 2>&1 < /dev/null & + SRV_PID=$! + for i in $(seq 1 180); do + curl -sf "http://127.0.0.1:$port/health" > /dev/null 2>&1 && { ok=1; break; } + kill -0 "$SRV_PID" 2>/dev/null || break + sleep 1 + done + if [ "$ok" = 1 ]; then + curl -sf "http://127.0.0.1:$port/props" > "$d/props.json" 2>/dev/null + THINKING_CAPABLE=$(python3 -c 'import json,sys +t = json.load(open(sys.argv[1])).get("chat_template") or "" +print("true" if ("enable_thinking" in t or "" in t) else "false")' "$d/props.json" 2>/dev/null || echo unknown) + for pid in $PIDS; do + curl -sf -X POST "http://127.0.0.1:$port/apply-template" -H 'Content-Type: application/json' \ + --data-binary @"$WORK/messages-$pid.json" > "$d/rendered-$pid.json" 2>/dev/null + python3 -c 'import json,sys +r = json.load(open(sys.argv[1])) +json.dump({"content": r["prompt"], "add_special": True, "parse_special": True}, open(sys.argv[2], "w"))' \ + "$d/rendered-$pid.json" "$d/tokreq-$pid.json" 2>/dev/null || continue + curl -sf -X POST "http://127.0.0.1:$port/tokenize" -H 'Content-Type: application/json' \ + --data-binary @"$d/tokreq-$pid.json" > "$d/ids-$pid.json" 2>/dev/null || continue + python3 - "$MANIFEST" "$SHA" "$pid" "$d/rendered-$pid.json" "$d/ids-$pid.json" <<'PY' +import json, sys +m, sha, pid, rendered, ids = sys.argv[1:6] +open(m, "a").write(json.dumps({"kind": "tok", "engine": "llama.cpp", "model_sha256": sha, + "prompt_id": pid, "rendered": rendered, "ids": ids}) + "\n") +PY + done + fi + kill "$SRV_PID" 2>/dev/null; wait "$SRV_PID" 2>/dev/null; SRV_PID="" + [ "$ok" = 1 ] +} + +# ---- the run --------------------------------------------------------------------- +printf -- '--- CRUX inference dogfood %s on %s (%s lane) ---\n' "$VERSION" "$HOST" "$BACKEND" +printf ' apr %s\n' "$APR_VERSION_LINE" +printf ' llama.cpp %s\n' "$( [ "$LLAMA_OK" = 1 ] && printf '%s' "${LLAMA_BUILD:-?}" || printf 'UNAVAILABLE: %s' "${LLAMA_WHY:-not requested}")" +printf ' ollama %s\n' "$( [ "$OLLAMA_OK" = 1 ] && printf 'server %s (client %s)' "$OLLAMA_SERVER" "${OLLAMA_CLIENT:-?}" || printf 'UNAVAILABLE: %s' "${OLLAMA_WHY:-not requested}")" + +# -dev none keeps the cpu lane's llama.cpp off every device, op-offload included; +# -ngl 0 alone still offloads large host ops to the GPU by default. +case "$BACKEND" in + gpu) APR_BE="--gpu"; NGL=999; LLAMA_DEV=() ;; + cpu) APR_BE="--no-gpu"; NGL=0; LLAMA_DEV=(-dev none) ;; +esac +VERB=run +for M_IN in "${MODELS[@]}"; do + M=$(readlink -f "$M_IN" 2>/dev/null) || M="" + [ -n "$M" ] && [ -f "$M" ] || decline "model not found: $M_IN" + SHA=$(sha256sum "$M" | cut -d' ' -f1) + SHA12=${SHA:0:12} + NAME=$(basename "$M") + mkdir -p "$WORK/$SHA12" + printf ' model %s sha256 %s\n' "$NAME" "$SHA" + + THINKING_CAPABLE=unknown + TOK_WHY="" + if [ "$LLAMA_OK" = 1 ]; then + llama_tokenize_prompts || TOK_WHY="llama.cpp server did not come up for tokenization (see its log)" + fi + + # ollama: import THIS file, then read back what ollama made of it. The name is + # per run: two lanes on one host must not `ollama rm` each other's import. + OL_NAME="crux-$SHA12-$BACKEND-$$" + OL_REFUSED="" + OL_BLOB="" + OL_TEMPLATE_SHA="" + if [ "$OLLAMA_OK" = 1 ]; then + { + printf 'FROM %s\n' "$M" + printf 'PARAMETER temperature %s\nPARAMETER seed %s\nPARAMETER num_predict %s\nPARAMETER num_ctx %s\n' "$TEMP" "$SEED" "$MAXTOK" "$CTX" + [ "$BACKEND" = cpu ] && printf 'PARAMETER num_gpu 0\n' + } > "$WORK/$SHA12/Modelfile" + "$OLLAMA" create "$OL_NAME" -f "$WORK/$SHA12/Modelfile" > "$WORK/$SHA12/ollama-create.log" 2>&1 + rc=$? + if [ "$rc" -ne 0 ]; then + OL_REFUSED="ollama create failed (rc $rc): $(tail -1 "$WORK/$SHA12/ollama-create.log" | tr -d '\r' | cut -c1-160)" + else + "$OLLAMA" show "$OL_NAME" --template > "$WORK/$SHA12/ollama-template.txt" 2>/dev/null + "$OLLAMA" show "$OL_NAME" --modelfile > "$WORK/$SHA12/ollama-modelfile.txt" 2>/dev/null + OL_BLOB=$(sed -n 's/^FROM .*sha256[-:]\([0-9a-f]\{64\}\).*/\1/p' "$WORK/$SHA12/ollama-modelfile.txt" | head -1) + OL_TEMPLATE_SHA=$(sha256sum "$WORK/$SHA12/ollama-template.txt" | cut -d' ' -f1) + # ollama 0.5.7 imports a GGUF with TEMPLATE {{ .Prompt }}: no chat template + # at all, so it would complete the raw prompt while the others chat. That is + # not an answer to the same question, and it is refused by name. + if [ "$(tr -d ' \n' < "$WORK/$SHA12/ollama-template.txt")" = "{{.Prompt}}" ]; then + OL_REFUSED="ollama $OLLAMA_SERVER imported the GGUF with no chat template (TEMPLATE {{ .Prompt }}), so it would complete the raw text rather than chat" + fi + fi + fi + + python3 - "$MODELS_JSONL" "$NAME" "$SHA" "$THINKING_CAPABLE" "$OL_BLOB" "$OL_TEMPLATE_SHA" "$OL_REFUSED" "$TOK_WHY" <<'PY' +import json, sys +f, name, sha, think, blob, tsha, oref, twhy = sys.argv[1:9] +open(f, "a").write(json.dumps({"name": name, "sha256": sha, + "thinking_capable": {"true": True, "false": False}.get(think), + "ollama": {"blob_sha256": blob or None, "blob_is_input": (blob == sha) if blob else None, + "template_sha256": tsha or None, "refused": oref or None}, + "tokenization": {"refused": twhy or None}}) + "\n") +PY + + # Thinking OFF only in this slice. apr has no toggle yet (#3723), and a + # thinking-capable model is told OFF explicitly by the comparators, which + # otherwise default to thinking. + THINK=off + LLAMA_THINK=() + OLLAMA_THINK=() + if [ "$THINKING_CAPABLE" = true ]; then + LLAMA_THINK=(--reasoning off) + if [ "$OLLAMA_OK" = 1 ]; then + ol_help=$("$OLLAMA" run --help 2>&1) + case "$ol_help" in *--think*) OLLAMA_THINK=(--think=false) ;; esac + fi + fi + + for pid in $PIDS; do + content=$(cat "$WORK/prompt-$pid.txt") + d="$WORK/$SHA12/$VERB" + mkdir -p "$d" + + if ! cell_lock; then + ref="the GPU lock was not had within ${LOCK_WAIT}s" + emit_gen apr "$pid" "" "" "" "$ref" + want llama.cpp && emit_gen llama.cpp "$pid" "" "" "" "$ref" + want ollama && emit_gen ollama "$pid" "" "" "" "$ref" + continue + fi + + eng_run "$d/apr-$pid.out" "$d/apr-$pid.err" "$APR" run "$M" --prompt "$content" --max-tokens "$MAXTOK" \ + --temperature "$TEMP" --seed "$SEED" --format json -v "$APR_BE" + rc=$? + emit_gen apr "$pid" "$rc" "$d/apr-$pid.out" "$d/apr-$pid.err" "" + + if [ "$LLAMA_OK" = 1 ]; then + eng_run "$d/llama-$pid.out" "$d/llama-$pid.err" "$LLAMA_CLI" -m "$M" -p "$content" -st -n "$MAXTOK" \ + --temp "$TEMP" --seed "$SEED" -c "$CTX" -ngl "$NGL" "${LLAMA_DEV[@]}" "${LLAMA_THINK[@]}" + rc=$? + emit_gen llama.cpp "$pid" "$rc" "$d/llama-$pid.out" "$d/llama-$pid.err" "" + elif want llama.cpp; then + emit_gen llama.cpp "$pid" "" "" "" "$LLAMA_WHY" + fi + + if [ "$OLLAMA_OK" = 1 ] && [ -z "$OL_REFUSED" ]; then + eng_run "$d/ollama-$pid.out" "$d/ollama-$pid.err" "$OLLAMA" run "$OL_NAME" "$content" --verbose --nowordwrap \ + --keepalive 0 "${OLLAMA_THINK[@]}" + rc=$? + unl=$(ollama_unloaded "$OL_NAME") + emit_gen ollama "$pid" "$rc" "$d/ollama-$pid.out" "$d/ollama-$pid.err" "" "$unl" + elif want ollama; then + emit_gen ollama "$pid" "" "" "" "${OL_REFUSED:-$OLLAMA_WHY}" + fi + cell_unlock + done + + if [ "$OLLAMA_OK" = 1 ] && [ "$KEEP_OLLAMA" = 0 ]; then + "$OLLAMA" rm "$OL_NAME" > /dev/null 2>&1 + fi +done + +# ---- judge -------------------------------------------------------------------------- +GPU_NAME=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | head -1) +HARNESS_SHA=$(git -C "$ROOT" rev-parse --short HEAD 2>/dev/null) || HARNESS_SHA="" +python3 - "$WORK/meta.json" "$MODELS_JSONL" "$VERSION" "$HOST" "$BACKEND" "$ENGINES" "$VERBS" \ + "$APR_VERSION_LINE" "${LLAMA_BUILD:-}" "$(llama_pin_get build_commit 2>/dev/null)" "$LLAMA_WHY" \ + "$OLLAMA_SERVER" "$OLLAMA_CLIENT" "$OLLAMA_WHY" "$TEMP" "$SEED" "$CTX" "$MAXTOK" "${GPU_NAME:-}" "$HARNESS_SHA" <<'PY' +import json, platform, sys +(out, models, version, host, backend, engines, verbs, apr_line, lbuild, lpin, lwhy, + osrv, ocli, owhy, temp, seed, ctx, maxtok, gpu, hsha) = sys.argv[1:21] +meta = { + "version": version, "host": host, "backend": backend, "isa": platform.machine(), "gpu": gpu or None, + "engines": engines.split(","), "verbs": verbs.split(","), "thinking": ["off"], + "sampling": {"temperature": float(temp), "seed": int(seed), "context": int(ctx), "max_tokens": int(maxtok), + "source": "scripts/llama_pin.toml [protocol]"}, + "apr": {"version_line": apr_line}, + "harness": {"sha": hsha or None, "driver": "scripts/crux_inference_dogfood.sh"}, + "llama_cpp": {"build": lbuild or None, "pin": lpin or None, "unavailable": lwhy or None}, + "ollama": {"server_version": osrv or None, "client_version": ocli or None, "unavailable": owhy or None}, + "models": [json.loads(l) for l in open(models) if l.strip()], + "not_covered": [ + "verbs chat, serve and code (the next slices of #3739)", + "thinking ON (apr has no toggle until #3723)", + "consumer-brief context rungs (#3716) and each engine's max accepted context", + "TTFT, which belongs to the serve verb's single OpenAI client", + ], +} +json.dump(meta, open(out, "w"), indent=2) +PY +mkdir -p "$OUT_DIR" || decline "cannot create $OUT_DIR" +python3 scripts/lib/crux_inference_judge.py collect --manifest "$MANIFEST" --prompts "$PROMPTS" \ + --meta "$WORK/meta.json" --out-json "$OUT_DIR/$HOST-$BACKEND.json" --out-md "$OUT_DIR/$HOST-$BACKEND.md" +rc=$? +printf 'receipt: %s/%s-%s.json (judge rc %s)\n' "$OUT_DIR" "$HOST" "$BACKEND" "$rc" +exit "$rc" diff --git a/scripts/crux_inference_prompts.json b/scripts/crux_inference_prompts.json new file mode 100644 index 0000000000..c8fdc73f78 --- /dev/null +++ b/scripts/crux_inference_prompts.json @@ -0,0 +1,27 @@ +{ + "schema": "crux-inference-prompts/v1", + "source": "crates/apr-cli/src/commands/golden_output.rs::golden_test_cases, the same three cases given as USER MESSAGES rather than pre-rendered ChatML: each engine applies the chat template it reads from the GGUF, and that step is part of what is under test (#3672 is a template defect a pre-rendered prompt would hide). scripts/check_crux_inference_judge.sh refuses drift between the two lists.", + "control": "golden-2plus2 is the POSITIVE CONTROL (the cop's ruling on #3739): the smallest model answers it right on every engine, so a control cell that comes back ALL_WRONG declines the run as a broken harness rather than recording a model limitation.", + "max_tokens": 64, + "prompts": [ + { + "id": "golden-2plus2", + "rung": "golden", + "control": true, + "messages": [{"role": "user", "content": "What is 2+2?"}], + "expect_any": ["4"] + }, + { + "id": "golden-greeting", + "rung": "golden", + "messages": [{"role": "user", "content": "Hello there, how are you doing today my friend?"}], + "expect_any": ["Hello", "Hi", "hey", "hello", "well", "!"] + }, + { + "id": "golden-paris", + "rung": "golden", + "messages": [{"role": "user", "content": "What is the capital of France?"}], + "expect_any": ["Paris"] + } + ] +} diff --git a/scripts/lib/crux_inference_judge.py b/scripts/lib/crux_inference_judge.py new file mode 100644 index 0000000000..d65462a48d --- /dev/null +++ b/scripts/lib/crux_inference_judge.py @@ -0,0 +1,431 @@ +#!/usr/bin/env python3 +"""crux_inference_judge.py: the judge of the CRUX inference dogfood (#3739). + +scripts/crux_inference_dogfood.sh DRIVES the engines and writes one manifest +line per invocation, pointing at that invocation's raw stdout/stderr. This file +READS those files and nothing else: it starts no engine and opens no socket. +The split is deliberate. The judge is the part the case table +(scripts/check_crux_inference_judge.sh) can drive with fixtures, with no model, +GPU or comparator, in milliseconds. + +THE ONE RULE (issue #3739 done_when 3). A cell is keyed by (model sha256, host, +verb, thinking, rung, prompt). It is RED when llama.cpp OR ollama answered it +correctly and apr did not. "apr did not" includes every way of not answering: +a non-zero exit, a backend that fell back to one the lane did not ask for, a +refusal, output the judge cannot parse, and an apr row that is simply MISSING. +Absence is a violation here, never a skip. + +The other outcomes: + GREEN apr correct, and at least one comparator ANSWERED (right or wrong) + UNJUDGED no comparator answered, so there is no external oracle for the cell + ALL_WRONG the comparators answered, none correctly, and apr was wrong too +ANY UNJUDGED cell declines the run (exit 2): it was never compared, and the +amended scope makes a missing cell a NO-GO. + +ALL_WRONG is NAMED, never a violation and never a pass (the cop's ruling, #3739: +the bar is "apr right wherever a competitor is right"). Two things keep it from +becoming the escape: the receipt counts ALL_WRONG per model, and the prompt set +must declare a POSITIVE CONTROL (`"control": true`), a prompt the smallest model +answers right on every engine. Every model must have a measured control cell. +A control cell that comes back ALL_WRONG is a broken harness, not a model +limitation, and declines the run; so does a model with no control cell, and a +prompt set that declares no control at all. + +PERFORMANCE IS TRANSCRIBED, NEVER COMPUTED. Token counts and rates are copied +from what each engine prints about itself, labelled with the engine that +reported them, and never ratioed, averaged or judged (the withdrawn-headline +rule, docs/BEATS.md). This file derives no rate from a clock, and no field it +writes is an input to the verdict. + +Usage: + crux_inference_judge.py collect --manifest M --prompts P --meta META \\ + --out-json R.json --out-md R.md +Exit: 0 no RED, no UNJUDGED, every model's control measured and not ALL_WRONG; +1 any RED; 2 decline. +""" + +import argparse +import datetime +import json +import re +import sys + +ENGINES = ("apr", "llama.cpp", "ollama") +COMPARATORS = ("llama.cpp", "ollama") +ANSI = re.compile(r"\x1b\[[0-9;?]*[A-Za-z]|\x1b\][^\x07]*\x07|\r") + + +def read_text(path): + if not path: + return "" + try: + with open(path, encoding="utf-8", errors="replace") as fh: + return fh.read() + except OSError: + return "" + + +def norm(text): + return " ".join(text.split()) + + +# ---------------------------------------------------------------- parsers -- +# Each parser returns {"answer": str|None, "why": str|None, "reported": {...}, +# plus engine-specific fields}. answer=None means "did not answer" and `why` +# says what the judge saw instead. + + +def parse_apr(stdout, stderr): + """apr's `run` verb with `--format json -v`: the answer is stdout JSON; -v + puts the prompt apr actually built, and its token ids, on stderr.""" + out = {"answer": None, "why": None, "reported": {}, "backend": None, + "prompt_ids": None, "prompt_token_count": None, "rendered_prompt": None} + m = re.search(r'formatted_prompt="((?:[^"\\]|\\.)*)"', stderr) + if m: + out["rendered_prompt"] = m.group(1) + m = re.search(r"encoded (\d+) tokens: \[([0-9, ]*)\]", stderr) + if m: + out["prompt_token_count"] = int(m.group(1)) + out["prompt_ids"] = [int(x) for x in m.group(2).replace(" ", "").split(",") if x] + start = stdout.find("{") + if start < 0: + out["why"] = "no JSON object on stdout" + return out + try: + doc, _ = json.JSONDecoder().raw_decode(stdout[start:]) + except ValueError as exc: + out["why"] = "stdout JSON unparseable: %s" % exc + return out + out["backend"] = doc.get("backend") + out["reported"] = { + "reported_by": "apr", + "prompt_tokens": out["prompt_token_count"], + "completion_tokens": doc.get("tokens_generated"), + "decode_rate_tokens_per_second": doc.get("tok_per_sec"), + "inference_ms": doc.get("inference_time_ms"), + } + text = doc.get("text") + if not isinstance(text, str): + out["why"] = "stdout JSON has no text field" + return out + out["answer"] = text + return out + + +RATE_LINE = re.compile(r"\[\s*Prompt:\s*([0-9.]+)\s*t/s\s*\|\s*Generation:\s*([0-9.]+)\s*t/s\s*\]") + + +def parse_llamacpp_cli(stdout, prompt_text): + """The pinned llama.cpp chat CLI echoes `> `, streams the answer, + then prints `[ Prompt: X t/s | Generation: Y t/s ]`. The answer is the text + between the echoed prompt and that line, and nothing else is guessed.""" + out = {"answer": None, "why": None, "reported": {}} + text = ANSI.sub("", stdout) + echo = "> " + prompt_text + at = text.rfind(echo) + if at < 0: + out["why"] = "the echoed prompt was not found in stdout" + return out + rest = text[at + len(echo):] + m = RATE_LINE.search(rest) + if not m: + out["why"] = "no end-of-turn timing line after the echoed prompt" + return out + out["answer"] = rest[:m.start()].strip() + out["reported"] = { + "reported_by": "llama.cpp", + "prompt_rate_tokens_per_second": float(m.group(1)), + "decode_rate_tokens_per_second": float(m.group(2)), + "prompt_tokens": None, + "completion_tokens": None, + } + return out + + +OLLAMA_STAT = re.compile(r"^(total duration|load duration|prompt eval count|prompt eval duration|" + r"prompt eval rate|eval count|eval duration|eval rate):\s*(.+?)\s*$", re.M) + + +def parse_ollama(stdout, stderr): + out = {"answer": None, "why": None, "reported": {}} + stats = {k: v for k, v in OLLAMA_STAT.findall(ANSI.sub("", stderr))} + + def num(key): + m = re.match(r"([0-9.]+)", stats.get(key, "")) + return float(m.group(1)) if m else None + + pe, ev = num("prompt eval count"), num("eval count") + out["reported"] = { + "reported_by": "ollama", + "prompt_tokens": int(pe) if pe is not None else None, + "completion_tokens": int(ev) if ev is not None else None, + "prompt_rate_tokens_per_second": num("prompt eval rate"), + "decode_rate_tokens_per_second": num("eval rate"), + "load_duration": stats.get("load duration"), + "total_duration": stats.get("total duration"), + } + answer = ANSI.sub("", stdout).strip() + if not answer: + out["why"] = "empty stdout" + return out + out["answer"] = answer + return out + + +# ------------------------------------------------------------------ judge -- + + +def correct(entry, expect_any): + return entry.get("answered", False) and any(p in (entry.get("answer") or "") for p in expect_any) + + +def engine_entry(row, prompt): + """One engine's answer to one cell, from its manifest row.""" + e = {"answered": False, "rc": row.get("rc"), "why": None, "answer": None, "reported": {}} + if "ollama_unloaded" in row: + # Did ollama's model leave VRAM before the cell dropped the GPU lock? + # Recorded, never judged: it is a property of the harness, not an answer. + e["vram_released"] = row["ollama_unloaded"] + if row.get("refused"): + e["why"] = "refused: " + row["refused"] + return e + stdout, stderr = read_text(row.get("stdout")), read_text(row.get("stderr")) + content = prompt["messages"][-1]["content"] + engine = row["engine"] + if engine == "apr": + p = parse_apr(stdout, stderr) + e["backend"] = p["backend"] + e["prompt_ids"] = p["prompt_ids"] + e["prompt_token_count"] = p["prompt_token_count"] + e["rendered_prompt"] = p["rendered_prompt"] + elif engine == "llama.cpp": + p = parse_llamacpp_cli(stdout, content) + elif engine == "ollama": + p = parse_ollama(stdout, stderr) + else: + e["why"] = "unknown engine %r" % engine + return e + e["reported"] = p["reported"] + e["answer"] = p["answer"] + if row.get("rc") != 0: + e["why"] = "exit %s" % row.get("rc") + return e + if p["answer"] is None: + e["why"] = p["why"] + return e + be = p.get("backend") if engine == "apr" else None + if be and (be.get("fell_back") or (row.get("backend") and be.get("ran") != row.get("backend"))): + e["why"] = "backend: asked %s, ran %s (fell_back=%s)" % (row.get("backend"), be.get("ran"), be.get("fell_back")) + return e + e["answered"] = True + return e + + +def token_parity(apr_entry, tok_row): + """apr's prompt ids (from -v) against llama.cpp's ids for ITS rendering of + the same messages with the GGUF's own template. apr prints at most a prefix + of its ids, so the comparison is the count plus that prefix, and says so.""" + if tok_row is None: + return {"measured": False, "why": "no llama.cpp tokenization row"} + try: + with open(tok_row["ids"], encoding="utf-8") as fh: + ref = json.load(fh).get("tokens") + except (OSError, ValueError, KeyError, TypeError) as exc: + return {"measured": False, "why": "llama.cpp ids unreadable: %s" % exc} + if not isinstance(ref, list): + return {"measured": False, "why": "llama.cpp ids missing"} + ids, n = apr_entry.get("prompt_ids"), apr_entry.get("prompt_token_count") + if ids is None or n is None: + return {"measured": False, "why": "apr printed no prompt ids", "llama_cpp_count": len(ref)} + first = next((i for i, (a, b) in enumerate(zip(ids, ref)) if a != b), None) + if first is None and len(ids) > len(ref): + first = len(ref) + return { + "measured": True, + "apr_count": n, + "llama_cpp_count": len(ref), + "apr_ids_compared": len(ids), + "apr_ids_complete": len(ids) == n, + "first_divergence": first, + "parity": n == len(ref) and first is None, + } + + +def judge_cell(entries, expect_any): + ok = {k: correct(v, expect_any) for k, v in entries.items()} + answered = {k: v.get("answered", False) for k, v in entries.items()} + if any(ok.get(c) for c in COMPARATORS) and not ok.get("apr"): + return "RED", ok + if not any(answered.get(c) for c in COMPARATORS): + return "UNJUDGED", ok + if ok.get("apr"): + return "GREEN", ok + return "ALL_WRONG", ok + + +def collect(args): + with open(args.prompts, encoding="utf-8") as fh: + prompts = {p["id"]: p for p in json.load(fh)["prompts"]} + with open(args.meta, encoding="utf-8") as fh: + meta = json.load(fh) + rows = [] + with open(args.manifest, encoding="utf-8") as fh: + for line in fh: + if line.strip(): + rows.append(json.loads(line)) + gens = [r for r in rows if r.get("kind") == "gen"] + toks = {(r["model_sha256"], r["prompt_id"]): r for r in rows if r.get("kind") == "tok"} + requested = meta.get("engines", list(ENGINES)) + + keys = [] + by_key = {} + for r in gens: + k = (r["model_sha256"], r["host"], r["verb"], r["thinking"], prompts[r["prompt_id"]]["rung"], r["prompt_id"]) + if k not in by_key: + keys.append(k) + by_key[k] = {} + by_key[k][r["engine"]] = r + + cells = [] + for k in keys: + prompt = prompts[k[5]] + entries = {} + for eng in ENGINES: + row = by_key[k].get(eng) + if row is None: + entries[eng] = {"answered": False, "missing": True, + "why": "missing: no row for this engine" if eng in requested else "not requested"} + else: + entries[eng] = engine_entry(row, prompt) + verdict, ok = judge_cell(entries, prompt["expect_any"]) + for eng in ENGINES: + entries[eng]["correct"] = ok[eng] + said = {e: norm(v["answer"]) for e, v in entries.items() if v.get("answered")} + cells.append({ + "key": dict(zip(("model_sha256", "host", "verb", "thinking", "rung", "prompt_id"), k)), + "verdict": verdict, + "expect_any": prompt["expect_any"], + "engines": entries, + "agreement": { + "answered": sorted(said), + "all_identical": len(said) >= 2 and len(set(said.values())) == 1, + "apr_matches": sorted(e for e in said if e != "apr" and "apr" in said and said[e] == said["apr"]), + }, + "token_parity": token_parity(entries["apr"], toks.get((k[0], k[5]))), + }) + + counts = {v: sum(1 for c in cells if c["verdict"] == v) for v in ("RED", "GREEN", "UNJUDGED", "ALL_WRONG")} + judged = counts["RED"] + counts["GREEN"] + counts["ALL_WRONG"] + all_wrong_by_model = {} + for c in cells: + if c["verdict"] == "ALL_WRONG": + k = c["key"]["model_sha256"] + all_wrong_by_model[k] = all_wrong_by_model.get(k, 0) + 1 + controls = [pid for pid, p in prompts.items() if p.get("control")] + broken = sorted({"%s/%s" % (c["key"]["model_sha256"][:12], c["key"]["prompt_id"]) + for c in cells if c["key"]["prompt_id"] in controls and c["verdict"] == "ALL_WRONG"}) + models = sorted({c["key"]["model_sha256"] for c in cells}) + uncontrolled = [m[:12] for m in models + if not any(c["key"]["model_sha256"] == m and c["key"]["prompt_id"] in controls for c in cells)] + declined_because = None + if not controls: + declined_because = "the prompt set declares no positive control (\"control\": true)" + elif not cells: + declined_because = "no cell was measured" + elif uncontrolled: + declined_because = "no positive-control cell was measured for model(s) " + ", ".join(uncontrolled) + elif broken: + declined_because = "positive control came back ALL_WRONG (a broken harness, not a model limit): " + ", ".join(broken) + if counts["RED"]: + verdict, rc = "RED", 1 + elif declined_because: + verdict, rc = "DECLINE", 2 + elif counts["UNJUDGED"]: + # An UNJUDGED cell was never compared to anything: under the amended + # scope (#3739, 17:19Z) "a missing cell is a NO-GO", and an unmeasured + # cell is a missing one. (A run with no GREEN cannot reach PASS: every + # model has a control cell, and a control that is not RED, UNJUDGED or + # ALL_WRONG is GREEN.) + verdict, rc = "DECLINE", 2 + else: + verdict, rc = "PASS", 0 + receipt = dict(meta) + receipt.update({ + "schema": "crux-inference-receipt/v1", + "judged_at": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%MZ"), + "cells": cells, + "summary": dict(counts, cells=len(cells), judged=judged, verdict=verdict, + all_wrong_by_model=all_wrong_by_model, controls=controls, + declined_because=declined_because if verdict == "DECLINE" else None), + }) + with open(args.out_json, "w", encoding="utf-8") as fh: + json.dump(receipt, fh, indent=2, ensure_ascii=False) + fh.write("\n") + md = render_md(receipt) + with open(args.out_md, "w", encoding="utf-8") as fh: + fh.write(md) + sys.stdout.write(md) + return rc + + +def short(text, n=48): + t = norm(text or "") + return (t[:n - 1] + "…") if len(t) > n else t + + +def render_md(r): + s = r["summary"] + lines = [ + "# CRUX inference dogfood: %s on %s (%s lane)" % (r.get("version"), r.get("host"), r.get("backend")), + "", + "apr `%s` · llama.cpp `%s` · ollama `%s` · judged %s" % ( + r.get("apr", {}).get("version_line"), r.get("llama_cpp", {}).get("build"), + r.get("ollama", {}).get("server_version"), r["judged_at"]), + "", + "**%s**: %d cells, %d RED, %d GREEN, %d ALL_WRONG, %d UNJUDGED." % ( + s["verdict"], s["cells"], s["RED"], s["GREEN"], s["ALL_WRONG"], s["UNJUDGED"]), + "", + "| model | verb | thinking | prompt | verdict | apr | llama.cpp | ollama | token parity |", + "|---|---|---|---|---|---|---|---|---|", + ] + names = {m["sha256"]: m.get("name", m["sha256"][:12]) for m in r.get("models", [])} + for c in r["cells"]: + k = c["key"] + cols = [] + for e in ENGINES: + v = c["engines"][e] + mark = "✅" if v["correct"] else ("❌" if v.get("answered") else "⛔") + cols.append("%s %s" % (mark, short(v["answer"]) if v.get("answered") else short(v.get("why"), 60))) + tp = c["token_parity"] + if tp.get("measured"): + tps = "%s (apr %d vs %d; first diff %s)" % ("=" if tp["parity"] else "≠", tp["apr_count"], + tp["llama_cpp_count"], tp["first_divergence"]) + else: + tps = "unmeasured: " + tp.get("why", "") + lines.append("| %s | %s | %s | %s | **%s** | %s | %s |" % ( + names.get(k["model_sha256"], k["model_sha256"][:12]), k["verb"], k["thinking"], k["prompt_id"], + c["verdict"], " | ".join(cols), tps)) + lines += ["", "✅ correct · ❌ answered, wrong · ⛔ did not answer (reason shown).", + "Rates and token counts are in the JSON, as each engine reported them, and are not judged.", ""] + nc = r.get("not_covered") + if nc: + lines += ["Not covered by this run (the issue requires them): " + "; ".join(nc) + ".", ""] + return "\n".join(lines) + + +def main(argv): + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + sub = ap.add_subparsers(dest="cmd", required=True) + c = sub.add_parser("collect") + for flag in ("--manifest", "--prompts", "--meta", "--out-json", "--out-md"): + c.add_argument(flag, required=True) + args = ap.parse_args(argv) + try: + return collect(args) + except (OSError, ValueError, KeyError) as exc: + sys.stderr.write("decline: %s: %s\n" % (type(exc).__name__, exc)) + return 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:]))