From 44f463360e4138e65f93915254c640f8dad658b6 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 21:07:02 +0200 Subject: [PATCH 1/5] =?UTF-8?q?PMAT-3774:=20evidence/crux/verb-corresponde?= =?UTF-8?q?nce.yaml=20=E2=80=94=20all=20253=20surface=20leaves,=204=20engi?= =?UTF-8?q?ne=20columns=20each=20stated=20(16=20verbs=20mapped,=20237=20no?= =?UTF-8?q?ne=20+=20reason),=20bound=20to=20surface=20sha256=20a7c6b7f1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refs #3774 Co-Authored-By: Claude Opus 5 (1M context) --- docs/roadmaps/entries/PMAT-3774.yaml | 21 + docs/roadmaps/roadmap.yaml | 21 + evidence/crux/verb-correspondence.yaml | 2745 ++++++++++++++++++++++++ 3 files changed, 2787 insertions(+) create mode 100644 docs/roadmaps/entries/PMAT-3774.yaml create mode 100644 evidence/crux/verb-correspondence.yaml diff --git a/docs/roadmaps/entries/PMAT-3774.yaml b/docs/roadmaps/entries/PMAT-3774.yaml new file mode 100644 index 0000000000..634431fa43 --- /dev/null +++ b/docs/roadmaps/entries/PMAT-3774.yaml @@ -0,0 +1,21 @@ +- id: PMAT-3774 + github_issue: 3774 + item_type: task + title: 'CRUX verb correspondence (#3739 amendments): every leaf of the derived apr surface maps to its llama.cpp / ollama / hf / llamafile counterpart, or `{none: reason}` per engine' + status: in_progress + priority: critical + assigned_to: aprender-76 + created: 2026-09-21T19:20:00Z + updated: 2026-09-21T19:40:00Z + spec: null + acceptance_criteria: + - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the surface emitted by apr 0.69.0 (621a47862, branch PMAT-3745-s1-surface-emitter), sha256 a7c6b7f1…6886. The entry key set equals the leaf key set, with no duplicates.' + - 'done_when 2, verbatim: "Every entry has `comparators` with ALL FOUR engine keys (`llama.cpp: {binary, args}`, `ollama: {subcommand, args}`, `hf: {api, args}`, `llamafile: {binary, args}`), each either that mapping or `{none: }`. An entry with at least one mapping has `judged_by` ∈ golden | greedy_tokens | task." MEASURED: all 253 entries state all 4 engines; 16 verbs map at least one engine (run, chat, serve run, code, bench, quantize, pull, list, rm, tokenize encode-corpus, eval, inspect, tensors, devices, modelfile parse, import); 237 are `none` on every engine, each with a reason.' + - 'done_when 3, verbatim: "Measured at authoring and recorded, each with a negative control that fails: every mapped llama.cpp `binary` exists in the pinned build (d1d3c3396) and every mapped flag appears in its `--help`; every ollama `subcommand` and flag appears in `ollama --help` (0.33.2); every mapped apr flag is an arg of that verb in the surface." MEASURED: 7/7 llama.cpp binaries and 10/10 flags ok on gx10''s pinned build; 7/7 ollama subcommands plus `create --quantize` ok on lambda''s 0.33.2; every mapped apr flag is present in its verb''s surface args. The controls `llama-cli --frobnicate`, bin `llama-frobnicate` and `ollama frobnicate` all FAIL.' + - 'done_when 4, out of scope, verbatim: "the `hf` and `llamafile` mappings are declared here but verified against their pins by the engine worker''s rows, since the pins don''t exist yet; the SHACL reading is aprender-97''s S2; the cells for newly mapped verbs are later #3739 slices."' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 44fc6b714f..f21c869340 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -19991,3 +19991,24 @@ roadmap: labels: - kind:code notes: 'Refs #3739, slice 1 (not a closing keyword; the issue has 6 done_when items and this meets parts of 1, 2 and 3). SCOPE AMENDMENT on the issue (issuecomment-5764587628, cop aprender-04, 17:19Z), verbatim: "Operator, verbatim: \"and CRUX for each verb in pre-release dogfood\". This applies the earlier operator line, \"the way to prevent testing leakage is tests are dervived from surface and SHACL enforced\" (#3745). The verb universe is DERIVED, not listed. It is every command in the surface the release-candidate binary emits (#3745 S1). The run/chat/serve/code list in this issue''s title is superseded as a scope; those four are the first slice, not the set. Every derived verb has a CRUX mapping entry in one committed correspondence file ... A derived verb with NO entry is a SHACL violation. Absence is never \"no comparator\". ... Pre-release, and it gates the tag. CRUX runs in the pre-publish dogfood (T-1), and its receipt is a `:CruxCell` input to `release-readiness-v1` (#3715/#3745). A cell where a comparator is right and apr is wrong is a NO-GO, and so is a missing cell." The cop''s ruling on sequencing, relayed the same minute: "The thin slice (qwen2.5-coder-1.5b `run`, lambda, 3 engines) stays first", and the correspondence file and `:CruxCell` fields are agreed with aprender-97 (#3745 S2, #3715) BEFORE they are coded. So this PR adds no correspondence file. The cop''s rulings of 18:47Z, verbatim: "(1) ALL_WRONG is NAMED, not a violation, since the bar is \"apr right wherever a competitor is right\". Two guards so it can''t become the escape: (i) it''s never scored Pass, and the receipt counts ALL_WRONG per model; (ii) every model''s prompt set includes a positive-control prompt that the smallest model answers right on every engine. If that control comes back ALL_WRONG, the run is DECLINED (rc 2) as a broken harness, never recorded as a model limitation." and "Every CRUX ollama request sets `keep_alive: 0`, runs under the GPU lock, and the cell checks VRAM is released before it drops the lock." Its ruling (2), the lambda ollama upgrade as a binary-only forjar change in paiml/infra, is slice 2 and is not in this PR. Its receipt already keys cells on the release-readiness tuple (model_sha256, host, verb, thinking, rung) plus prompt_id, as aprender-97 asked, and makes an UNJUDGED (unmeasured) cell decline. The issue''s done_when, verbatim: (1) "`scripts/crux_inference_dogfood.sh ` runs, per inventory Q4_K model × {lambda, gx10}, the SAME prompt set (the golden prompts + a consumer brief from #3716 at each declared context rung that fits) through: run: `apr run` · `llama-cli` · `ollama run`; chat (multi-turn): `apr chat` · llama-cli conversation · `ollama run` multi-turn; serve: `apr serve` · `llama-server` · `ollama serve`, all through the SAME OpenAI `/v1/chat/completions` client, streaming + non-streaming; code: `apr code` on a scripted edit-and-verify task, and the same task driven through llama-server and ollama by the same client; thinking ON and OFF where the model supports it (#3723), at temperature 0, same seed" (2) "per cell it records: answer correct (golden), exact-match agreement across engines, token-id parity (apr vs `llama-tokenize`), TTFT, decode tok/s, prompt/completion tokens, and the max context each engine accepts before refusing (or failing silently)" (3) "the gate: a cell where llama.cpp OR ollama answers correctly and apr doesn''t is RED (a finding naming the cell). Performance ratios are REPORTED, never claimed (the withdrawn-headline rule). The receipt feeds the SHACL shape (#3715) as a required cell kind." (4) "ollama is upgraded on lambda through the declared path (forjar) to a version that loads qwen35 and qwen3moe GGUFs; both hosts'' versions are recorded" (5) "runs in the release train (the T-1 `models` step, #3717) and ALSO on demand; the results table is committed per release (`evidence/crux//`)" (6) "depends on: #3563 (A), #3723, #3571". SLICING, ruled by the cop (aprender-04, 2026-09-21: "Start with a design note on #3739 and a first thin slice (qwen2.5-coder-1.5b Q4_K_M, `run` verb, lambda, all 3 engines) to prove the harness, then widen") and laid out in the design note: 1 run (this PR); 2 ollama on lambda via forjar + gx10; 3 chat; 4 serve through one client (perf via `apr test llm bench`, needs #3571); 5 code; 6 thinking ON (needs #3723); 7 consumer rungs + max context; 8 T-1 wiring (#3717). SCOPE: scripts/crux_inference_dogfood.sh (driver), scripts/lib/crux_inference_judge.py (judge), scripts/check_crux_inference_judge.sh (case table), scripts/crux_inference_prompts.json (prompt set), evidence/crux/0.69.0/*, this fragment. ONE DELIBERATE DEVIATION from the letter of done_when 2: token-id parity uses llama.cpp''s server `/tokenize` on its own `/apply-template` rendering, not the `llama-tokenize` binary. lambda''s pinned build dir (the one llama_bin.sh resolves) has no `llama-tokenize`, the server exists on both hosts, and both call the same llama_tokenize on the same vocab. Rendering first also makes parity cover the chat template, which is where #3672 lives. PERF-009: the driver starts engines but contains no rate token and reads no clock, and the judge starts nothing, so the competing-harness count stays 0 without an allowlist entry. The serve slice takes TTFT and decode rate from `apr test llm bench`, not from a new timer. FINDINGS on lambda, apr 0.69.0 @ 52f43da71, qwen2.5-coder-1.5b Q4_K_M sha256 cc324af0…: (a) apr renders the chat template twice (#3672): 46 prompt ids against llama.cpp''s 36, diverging at id 1, and the U+200B escapes encode as id 0 (#3726). The golden answer is still right ("4"), so only the parity field sees it. (b) ollama 0.5.7 imports a GGUF with no chat template; gx10''s 0.33.2 imports the GGUF''s own jinja template but stores blob b9a2f9f3…, not the input''s cc324af0…' +- id: PMAT-3774 + github_issue: 3774 + item_type: task + title: 'CRUX verb correspondence (#3739 amendments): every leaf of the derived apr surface maps to its llama.cpp / ollama / hf / llamafile counterpart, or `{none: reason}` per engine' + status: in_progress + priority: critical + assigned_to: aprender-76 + created: 2026-09-21T19:20:00Z + updated: 2026-09-21T19:40:00Z + spec: null + acceptance_criteria: + - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the surface emitted by apr 0.69.0 (621a47862, branch PMAT-3745-s1-surface-emitter), sha256 a7c6b7f1…6886. The entry key set equals the leaf key set, with no duplicates.' + - 'done_when 2, verbatim: "Every entry has `comparators` with ALL FOUR engine keys (`llama.cpp: {binary, args}`, `ollama: {subcommand, args}`, `hf: {api, args}`, `llamafile: {binary, args}`), each either that mapping or `{none: }`. An entry with at least one mapping has `judged_by` ∈ golden | greedy_tokens | task." MEASURED: all 253 entries state all 4 engines; 16 verbs map at least one engine (run, chat, serve run, code, bench, quantize, pull, list, rm, tokenize encode-corpus, eval, inspect, tensors, devices, modelfile parse, import); 237 are `none` on every engine, each with a reason.' + - 'done_when 3, verbatim: "Measured at authoring and recorded, each with a negative control that fails: every mapped llama.cpp `binary` exists in the pinned build (d1d3c3396) and every mapped flag appears in its `--help`; every ollama `subcommand` and flag appears in `ollama --help` (0.33.2); every mapped apr flag is an arg of that verb in the surface." MEASURED: 7/7 llama.cpp binaries and 10/10 flags ok on gx10''s pinned build; 7/7 ollama subcommands plus `create --quantize` ok on lambda''s 0.33.2; every mapped apr flag is present in its verb''s surface args. The controls `llama-cli --frobnicate`, bin `llama-frobnicate` and `ollama frobnicate` all FAIL.' + - 'done_when 4, out of scope, verbatim: "the `hf` and `llamafile` mappings are declared here but verified against their pins by the engine worker''s rows, since the pins don''t exist yet; the SHACL reading is aprender-97''s S2; the cells for newly mapped verbs are later #3739 slices."' + phases: [] + subtasks: [] + estimated_effort: null + labels: + - kind:code + notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' diff --git a/evidence/crux/verb-correspondence.yaml b/evidence/crux/verb-correspondence.yaml new file mode 100644 index 0000000000..83ee461750 --- /dev/null +++ b/evidence/crux/verb-correspondence.yaml @@ -0,0 +1,2745 @@ +# evidence/crux/verb-correspondence.yaml (#3774, child of #3739). +# DERIVED from the surface named below; re-derive when the surface's sha256 changes. +# Schema agreed with aprender-97 (the shape side); columns hf and llamafile per the +# 19:03Z / 19:04Z amendments of #3739. Engine pins are the engine worker's rows. +schema: crux-verb-correspondence/v1 +surface: + source: 'apr surface --json (apr-cli-surface/v1, #3745 S1), emitted by apr 0.69.0 (621a47862)' + sha256: a7c6b7f1b66aaae231363904456bd0a6b13d98dfdfb5c9064b5f7407076e6886 + leaves: 253 +engines: +- llama.cpp +- ollama +- hf +- llamafile +rule: 'every leaf of the surface has exactly one entry; every entry states every engine, as a mapping or {none: }. A mapped engine is a required :CruxCell (#3715/#3745 S2 reads this file).' +verbs: +- key: attn-parity-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: attn-viz-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: audio-inspect-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: awq-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: beat-run + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: bench + judged_by: task + comparators: + llama.cpp: + binary: llama-bench + args: + --max-tokens: -n + ollama: + none: no benchmark verb; ollama's own rates come from `run --verbose` and are reported in the run cells + hf: + none: transformers ships no benchmark CLI; a hand-written timing loop would be a competing harness (PERF-009) + llamafile: + none: the llamafile pin (the engine worker's row) declares no benchmark mode; revisit if it does +- key: canary check + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: canary create + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: cbtop + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: cgp baseline + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp bench + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp compete + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp contract generate + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp contract verify + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp diff + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp doctor + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp explain + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile binary + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile compare + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile cublas + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile kernel + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile library + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile metal + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile parallel + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile python + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile quant + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile scalar + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile scaling + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile simd + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile wasm + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp profile wgpu + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp roofline + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp trace + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: cgp tui + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (cgp): a sibling tool mounted under apr, not an apr inference verb' +- key: chat + judged_by: golden + comparators: + llama.cpp: + binary: llama-cli + args: + --system: -sys + --max-tokens: -n + --temperature: --temp + ollama: + subcommand: run + args: {} + hf: + api: apply_chat_template+generate + args: + --max-tokens: max_new_tokens + llamafile: + binary: llamafile + args: + --system: -sys + --max-tokens: -n + --temperature: --temp +- key: check + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: check-finite-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: code + judged_by: task + comparators: + llama.cpp: + binary: llama-server + args: {} + ollama: + subcommand: serve + args: {} + hf: + api: transformers serve + args: {} + llamafile: + binary: llamafile + args: {} +- key: compare-hf + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: compile + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: convert + comparators: + llama.cpp: + none: APR-to-APR optimisation; no engine reads APR + ollama: + none: APR-to-APR optimisation; no engine reads APR + hf: + none: APR-to-APR optimisation; no engine reads APR + llamafile: + none: APR-to-APR optimisation; no engine reads APR +- key: data audit + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: data balance + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: data decontaminate + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: data dedup + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: data split + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: data x convert + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x dedup + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x drift compare + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x drift detect + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x drift merge + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x drift report + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x drift sketch + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x fed manifest + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x fed plan + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x fed split + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x fed verify + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x filter-text + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x fim + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x head + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x import local + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x info + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x mix + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x quality check + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x quality profiles + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x quality report + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x quality score + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x registry delete + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x registry init + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x registry list + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x registry pull + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x registry push + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x registry search + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x registry show-info + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x repl + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x schema + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: data x view + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (data x): a sibling tool mounted under apr, not an apr inference verb' +- key: dataset audio-inspect + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: ddp-metrics-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: debug embed-viz + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: decrypt + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: devices + judged_by: task + comparators: + llama.cpp: + binary: llama-cli + args: + : --list-devices + ollama: + none: ollama reports devices only in its server log, not as a verb + hf: + none: device discovery is torch's, not a transformers verb + llamafile: + none: llamafile has no device-listing verb at the pin (the engine worker's row measures it) +- key: diagnose + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: diff + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: distill + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: dry-sampling-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: embed + comparators: + llama.cpp: + none: embeds with an APR-format BERT bi-encoder; llama.cpp, ollama and transformers embed from their own formats, so no same-file comparison exists until apr embed reads GGUF/SafeTensors + ollama: + none: embeds with an APR-format BERT bi-encoder; llama.cpp, ollama and transformers embed from their own formats, so no same-file comparison exists until apr embed reads GGUF/SafeTensors + hf: + none: embeds with an APR-format BERT bi-encoder; llama.cpp, ollama and transformers embed from their own formats, so no same-file comparison exists until apr embed reads GGUF/SafeTensors + llamafile: + none: embeds with an APR-format BERT bi-encoder; llama.cpp, ollama and transformers embed from their own formats, so no same-file comparison exists until apr embed reads GGUF/SafeTensors +- key: embed-viz-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: embeddings-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: encrypt + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: eval + judged_by: task + comparators: + llama.cpp: + binary: llama-perplexity + args: + --text: -f + ollama: + none: ollama exposes no perplexity or log-likelihood verb + hf: + api: AutoModelForCausalLM(labels=...).loss + args: {} + llamafile: + none: llamafile's release binary has no perplexity mode +- key: experiment view + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: explain + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: explain-token-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: export + comparators: + llama.cpp: + none: exports APR to other formats; the comparators CONSUME the export (the GGUF under test), none of them is a competing exporter + ollama: + none: exports APR to other formats; the comparators CONSUME the export (the GGUF under test), none of them is a competing exporter + hf: + none: exports APR to other formats; the comparators CONSUME the export (the GGUF under test), none of them is a competing exporter + llamafile: + none: exports APR to other formats; the comparators CONSUME the export (the GGUF under test), none of them is a competing exporter +- key: finetune + comparators: + llama.cpp: + none: trains LoRA/QLoRA adapters; llama-finetune at the pin trains full weights (its --lora only loads an adapter), ollama and llamafile do not train + ollama: + none: trains LoRA/QLoRA adapters; llama-finetune at the pin trains full weights (its --lora only loads an adapter), ollama and llamafile do not train + hf: + none: trains LoRA/QLoRA adapters; llama-finetune at the pin trains full weights (its --lora only loads an adapter), ollama and llamafile do not train + llamafile: + none: trains LoRA/QLoRA adapters; llama-finetune at the pin trains full weights (its --lora only loads an adapter), ollama and llamafile do not train +- key: flow + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: fp8-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: gbnf-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: gptq-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: gpu + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: gpu-memtrace-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: grad-norm + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: hang-trace-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: hex + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: imatrix-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: import + judged_by: task + comparators: + llama.cpp: + none: llama.cpp imports through convert_hf_to_gguf.py, a Python script outside the pinned binaries + ollama: + subcommand: create + args: {} + hf: + none: HF is the SOURCE apr imports from, not a competing importer + llamafile: + none: llamafile has no import verb +- key: inspect + judged_by: task + comparators: + llama.cpp: + binary: llama-gguf + args: {} + ollama: + subcommand: show + args: {} + hf: + api: AutoConfig.from_pretrained + args: {} + llamafile: + none: llamafile has no model-metadata verb +- key: kernel parity + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: kv-timeline-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: lint + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: list + aliases: + - ls + judged_by: task + comparators: + llama.cpp: + none: llama.cpp keeps no model cache to list + ollama: + subcommand: list + args: {} + hf: + none: the HF cache is listed by `hf cache scan`, a huggingface_hub CLI, not transformers; not pinned by the engine row + llamafile: + none: llamafile keeps no model cache to list +- key: manifest + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: mcp + comparators: + llama.cpp: + none: apr's MCP server; no engine ships an MCP server + ollama: + none: apr's MCP server; no engine ships an MCP server + hf: + none: apr's MCP server; no engine ships an MCP server + llamafile: + none: apr's MCP server; no engine ships an MCP server +- key: merge + comparators: + llama.cpp: + none: merges whole models (weighted/slerp); llama-export-lora only merges a LoRA into its base + ollama: + none: merges whole models (weighted/slerp); llama-export-lora only merges a LoRA into its base + hf: + none: merges whole models (weighted/slerp); llama-export-lora only merges a LoRA into its base + llamafile: + none: merges whole models (weighted/slerp); llama-export-lora only merges a LoRA into its base +- key: modelfile parse + judged_by: task + comparators: + llama.cpp: + none: Modelfiles are ollama's format; llama.cpp does not read them + ollama: + subcommand: create + args: {} + hf: + none: Modelfiles are ollama's format; transformers does not read them + llamafile: + none: Modelfiles are ollama's format; llamafile does not read them +- key: monitor + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: nccl-diag-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: nf4-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: ollama-chat-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: ollama-tools-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: oom-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: oracle + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: otlp-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: parity + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: pipeline apply + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: pipeline plan + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: pipeline status + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: pipeline validate + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: ppl + comparators: + llama.cpp: + none: a perplexity-based CLASSIFIER (CRUX-E-02), not a perplexity measurement; no engine classifies + ollama: + none: a perplexity-based CLASSIFIER (CRUX-E-02), not a perplexity measurement; no engine classifies + hf: + none: a perplexity-based CLASSIFIER (CRUX-E-02), not a perplexity measurement; no engine classifies + llamafile: + none: a perplexity-based CLASSIFIER (CRUX-E-02), not a perplexity measurement; no engine classifies +- key: pretrain + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: profile + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: prometheus-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: prune + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: ptx + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: ptx-map + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: publish + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: pull + judged_by: task + comparators: + llama.cpp: + none: llama.cpp downloads a model only as a side effect of running it (`-hf`), not as a verb + ollama: + subcommand: pull + args: {} + hf: + api: huggingface_hub.snapshot_download + args: + --revision: revision + --include: allow_patterns + llamafile: + none: llamafile has no model registry or download verb +- key: pv audit + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv book + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv census + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv certify + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv check-parity + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv codegen + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv coq + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv coverage + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv diff + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv equations + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv explain + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv extract + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv extract-pytorch + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv flux + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv fuzz + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv generate + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv graph + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv infer + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv invariants + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv kaizen + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv kani + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv lean + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv lean-status + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv lint + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv migrate + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv mirai + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv pipeline + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv probar + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv proof-status + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv query + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv roofline + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv scaffold + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv score + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv status + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv tla + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv unlock + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv validate + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv verify-bindings + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv verify-pipeline + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: pv verify-structure + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (pv): a sibling tool mounted under apr, not an apr inference verb' +- key: qa + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: qualify + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: quant-preservation-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: quantize + judged_by: task + comparators: + llama.cpp: + binary: llama-quantize + args: {} + ollama: + subcommand: create + args: + --scheme: --quantize + hf: + none: transformers quantizes through bitsandbytes/GPTQ backends, never to GGUF Q4_K, so there is no same-format output to compare + llamafile: + none: llamafile's release binary runs models; quantizing is not one of its modes +- key: rag demo + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' +- key: rag extract-frames + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' +- key: rag index + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' +- key: rag info + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' +- key: rag query + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' +- key: rag transcribe + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (rag): a sibling tool mounted under apr, not an apr inference verb' +- key: react-trace-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: registry aliases + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: registry-quota-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: rerank + comparators: + llama.cpp: + none: ranks with an APR-format BERT cross-encoder; no engine reranks the same file (llama-server's /rerank needs a GGUF reranker) + ollama: + none: ranks with an APR-format BERT cross-encoder; no engine reranks the same file (llama-server's /rerank needs a GGUF reranker) + hf: + none: ranks with an APR-format BERT cross-encoder; no engine reranks the same file (llama-server's /rerank needs a GGUF reranker) + llamafile: + none: ranks with an APR-format BERT cross-encoder; no engine reranks the same file (llama-server's /rerank needs a GGUF reranker) +- key: rm + aliases: + - remove + judged_by: task + comparators: + llama.cpp: + none: llama.cpp keeps no model cache to remove from + ollama: + subcommand: rm + args: {} + hf: + none: not a transformers verb (cache deletion lives in huggingface_hub, not pinned by the engine row) + llamafile: + none: llamafile keeps no model cache to remove from +- key: rm-gc-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: rosetta chain + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: rosetta compare-inference + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: rosetta convert + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: rosetta diff-tensors + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: rosetta fingerprint + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: rosetta inspect + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: rosetta validate-stats + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: rosetta verify + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: run + judged_by: golden + comparators: + llama.cpp: + binary: llama-cli + args: + --prompt: -p + --max-tokens: -n + --temperature: --temp + --seed: --seed + ollama: + subcommand: run + args: + --prompt: + hf: + api: AutoModelForCausalLM.generate + args: + --max-tokens: max_new_tokens + llamafile: + binary: llamafile + args: + --prompt: -p + --max-tokens: -n + --temperature: --temp + --seed: --seed +- key: runs diff + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: runs ls + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: runs show + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: serve plan + comparators: + llama.cpp: + none: a pre-flight capacity plan (VRAM budget, roofline); it runs no model + ollama: + none: a pre-flight capacity plan (VRAM budget, roofline); it runs no model + hf: + none: a pre-flight capacity plan (VRAM budget, roofline); it runs no model + llamafile: + none: a pre-flight capacity plan (VRAM budget, roofline); it runs no model +- key: serve run + judged_by: golden + comparators: + llama.cpp: + binary: llama-server + args: + --port: --port + --context-length: -c + ollama: + subcommand: serve + args: {} + hf: + api: transformers serve + args: + --port: --port + llamafile: + binary: llamafile + args: + --port: --port + --context-length: -c +- key: shard + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: shared-cache-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: showcase + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: sim emc-check + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' +- key: sim emc-validate + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' +- key: sim help + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' +- key: sim list-emc + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' +- key: sim render + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' +- key: sim run + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' +- key: sim validate + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' +- key: sim verify + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' +- key: sim version + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (sim): a sibling tool mounted under apr, not an apr inference verb' +- key: stamp + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: surface + hidden: true + comparators: + llama.cpp: + none: the surface emitter itself (hidden); it prints apr's own command tree + ollama: + none: the surface emitter itself (hidden); it prints apr's own command tree + hf: + none: the surface emitter itself (hidden); it prints apr's own command tree + llamafile: + none: the surface emitter itself (hidden); it prints apr's own command tree +- key: tensors + judged_by: task + comparators: + llama.cpp: + binary: llama-gguf + args: {} + ollama: + none: ollama exposes no tensor listing + hf: + api: safetensors.safe_open(...).keys + args: {} + llamafile: + none: llamafile has no tensor-listing verb +- key: test llm bench + comparators: + llama.cpp: + none: the OpenAI-client harness the serve cells use against every engine's server; it is the instrument, not a verb with a competitor + ollama: + none: the OpenAI-client harness the serve cells use against every engine's server; it is the instrument, not a verb with a competitor + hf: + none: the OpenAI-client harness the serve cells use against every engine's server; it is the instrument, not a verb with a competitor + llamafile: + none: the OpenAI-client harness the serve cells use against every engine's server; it is the instrument, not a verb with a competitor +- key: test tensor + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: tokenize apply + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: tokenize encode-corpus + judged_by: task + comparators: + llama.cpp: + binary: llama-tokenize + args: {} + ollama: + none: ollama exposes no tokenize verb or endpoint + hf: + api: AutoTokenizer + args: {} + llamafile: + none: llamafile tokenizes with its embedded, older llama.cpp; the llama.cpp column already tests the tokenizer at the current pin +- key: tokenize import-hf + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: tokenize plan + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: tokenize repair-manifest + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: tokenize train + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: tool-use-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: trace + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: train apply + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: train archive + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: train cluster-status + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: train halving + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: train plan + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: train submit + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: train sweep + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: train watch + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: tree + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: tui + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: tune + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: typical-p-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: unified-search-lint + comparators: + llama.cpp: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + ollama: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + hf: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart + llamafile: + none: lints an externally captured artifact; it runs no model, so no engine has a counterpart +- key: unshard + comparators: + llama.cpp: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + ollama: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + hf: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb + llamafile: + none: trains or transforms weights or data; CRUX compares inference behaviour (#3739), and llama.cpp, ollama and llamafile have no same-task verb +- key: validate + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: validate-manifest + comparators: + llama.cpp: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + ollama: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + hf: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + llamafile: + none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent +- key: zram benchmark + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' +- key: zram create + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' +- key: zram remove + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' +- key: zram status + foreign: true + comparators: + llama.cpp: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + ollama: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + hf: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' + llamafile: + none: 'foreign subtree (zram): a sibling tool mounted under apr, not an apr inference verb' From a8f8a3627d77834067040b3f404a91cfc9eda440 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 23:02:34 +0200 Subject: [PATCH 2/5] =?UTF-8?q?PMAT-3774:=20the=20judge=20side=20of=20the?= =?UTF-8?q?=20hf=20and=20llamafile=20engines=20=E2=80=94=20comparators,=20?= =?UTF-8?q?tok/tmpl=20byte-equal-or-RED,=20greedy=20reported,=20the=20plug?= =?UTF-8?q?in-engine=20driver=20hook;=20correspondence=20re-derived=20at?= =?UTF-8?q?=20surface=20v1.1=20(207c3297)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refs #3774 Co-Authored-By: Claude Opus 5 (1M context) --- docs/roadmaps/entries/PMAT-3774.yaml | 10 +- docs/roadmaps/roadmap.yaml | 10 +- evidence/crux/verb-correspondence.yaml | 41 +++-- scripts/check_crux_inference_judge.sh | 131 +++++++++++++++- scripts/crux_inference_dogfood.sh | 88 ++++++++++- scripts/lib/crux_inference_judge.py | 199 +++++++++++++++++++++++-- 6 files changed, 435 insertions(+), 44 deletions(-) diff --git a/docs/roadmaps/entries/PMAT-3774.yaml b/docs/roadmaps/entries/PMAT-3774.yaml index 634431fa43..3f71400578 100644 --- a/docs/roadmaps/entries/PMAT-3774.yaml +++ b/docs/roadmaps/entries/PMAT-3774.yaml @@ -6,16 +6,20 @@ priority: critical assigned_to: aprender-76 created: 2026-09-21T19:20:00Z - updated: 2026-09-21T19:40:00Z + updated: 2026-09-21T21:05:00Z spec: null acceptance_criteria: - - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the surface emitted by apr 0.69.0 (621a47862, branch PMAT-3745-s1-surface-emitter), sha256 a7c6b7f1…6886. The entry key set equals the leaf key set, with no duplicates.' + - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the `apr-cli-surface/v1.1` surface emitted by apr 0.69.0 (7ed033b63, branch PMAT-3745-s1-surface-emitter), sha256 207c3297…da2b. The entry key set equals the leaf key set, with no duplicates. Every `generates: true` leaf that maps no engine (embed, rerank, mcp, parity, rosetta compare-inference) has a specific reason; parity and compare-inference compare apr with apr, and their external comparison is the `run` cells.' - 'done_when 2, verbatim: "Every entry has `comparators` with ALL FOUR engine keys (`llama.cpp: {binary, args}`, `ollama: {subcommand, args}`, `hf: {api, args}`, `llamafile: {binary, args}`), each either that mapping or `{none: }`. An entry with at least one mapping has `judged_by` ∈ golden | greedy_tokens | task." MEASURED: all 253 entries state all 4 engines; 16 verbs map at least one engine (run, chat, serve run, code, bench, quantize, pull, list, rm, tokenize encode-corpus, eval, inspect, tensors, devices, modelfile parse, import); 237 are `none` on every engine, each with a reason.' - 'done_when 3, verbatim: "Measured at authoring and recorded, each with a negative control that fails: every mapped llama.cpp `binary` exists in the pinned build (d1d3c3396) and every mapped flag appears in its `--help`; every ollama `subcommand` and flag appears in `ollama --help` (0.33.2); every mapped apr flag is an arg of that verb in the surface." MEASURED: 7/7 llama.cpp binaries and 10/10 flags ok on gx10''s pinned build; 7/7 ollama subcommands plus `create --quantize` ok on lambda''s 0.33.2; every mapped apr flag is present in its verb''s surface args. The controls `llama-cli --frobnicate`, bin `llama-frobnicate` and `ollama frobnicate` all FAIL.' - 'done_when 4, out of scope, verbatim: "the `hf` and `llamafile` mappings are declared here but verified against their pins by the engine worker''s rows, since the pins don''t exist yet; the SHACL reading is aprender-97''s S2; the cells for newly mapped verbs are later #3739 slices."' + - 'Added to #3774 at ~19:30Z (the cop: "You own the correspondence file columns (`hf` and `llamafile`) and the judge side (#3774)"), verbatim: "`scripts/lib/crux_inference_judge.py` treats `hf` and `llamafile` as comparators: RED when either is right and apr is wrong." MEASURED by `bash scripts/check_crux_inference_judge.sh`: 33 ok, 0 broke. New rows cover: hf right with apr wrong is RED; a llamafile refusal is quoted and, beside a right llama.cpp, the cell is GREEN; hf output that is not the contract JSON, or JSON with no `text`, is no oracle.' + - 'Verbatim: "`tok` rows ... and `tmpl` rows ... are byte-equal or RED, with the first differing index or byte named. An engine row with no apr row beside it is RED. A row the reference engine refused is UNJUDGED, and the run declines. `greedy` rows are REPORTED." MEASURED rows: equal tok ids are GREEN; differing ids are RED at index 1; an HF tok row without apr is RED; an HF-refused tok row declines rc 2; an equal tmpl is GREEN; the #3672 double wrap is RED at byte 17; greedy divergence at step 2 is reported with its logit cosine 0.7071 and no verdict.' + - 'Verbatim: "`scripts/crux_inference_dogfood.sh` exports `CRUX_MANIFEST` and `CRUX_WORK`, resolves `scripts/crux_engine_.{sh,py}` ... A missing script, a failing probe, or a `gen` that appends no row becomes a refused row naming why." MEASURED on lambda with a throwaway fake hf driver (never committed): probe recorded verbatim as `hf.probe`; 3 hf gen rows judged correct; the absent llamafile driver refused by name on every cell; the receipt table has 5 engine columns. The driver still reads no clock (0 rate tokens) and check_no_competing_harnesses stays count=0.' + - 'Mutation for the judge side: dropping the tok/tmpl mismatch-or-absent RED breaks 3 rows; not failing the run on a deterministic RED breaks 3; not declining a deterministic UNJUDGED breaks 1; dropping hf as a comparator breaks 1; not requiring the contract `text` field breaks 1; taking the logit cosine at the wrong step breaks 1. The slice-1 mutants still hold.' phases: [] subtasks: [] estimated_effort: null labels: - kind:code - notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' + notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml, scripts/lib/crux_inference_judge.py, scripts/crux_inference_dogfood.sh (the plugin-engine hook), scripts/check_crux_inference_judge.sh, and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index f21c869340..579e61158d 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -19999,16 +19999,20 @@ roadmap: priority: critical assigned_to: aprender-76 created: 2026-09-21T19:20:00Z - updated: 2026-09-21T19:40:00Z + updated: 2026-09-21T21:05:00Z spec: null acceptance_criteria: - - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the surface emitted by apr 0.69.0 (621a47862, branch PMAT-3745-s1-surface-emitter), sha256 a7c6b7f1…6886. The entry key set equals the leaf key set, with no duplicates.' + - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the `apr-cli-surface/v1.1` surface emitted by apr 0.69.0 (7ed033b63, branch PMAT-3745-s1-surface-emitter), sha256 207c3297…da2b. The entry key set equals the leaf key set, with no duplicates. Every `generates: true` leaf that maps no engine (embed, rerank, mcp, parity, rosetta compare-inference) has a specific reason; parity and compare-inference compare apr with apr, and their external comparison is the `run` cells.' - 'done_when 2, verbatim: "Every entry has `comparators` with ALL FOUR engine keys (`llama.cpp: {binary, args}`, `ollama: {subcommand, args}`, `hf: {api, args}`, `llamafile: {binary, args}`), each either that mapping or `{none: }`. An entry with at least one mapping has `judged_by` ∈ golden | greedy_tokens | task." MEASURED: all 253 entries state all 4 engines; 16 verbs map at least one engine (run, chat, serve run, code, bench, quantize, pull, list, rm, tokenize encode-corpus, eval, inspect, tensors, devices, modelfile parse, import); 237 are `none` on every engine, each with a reason.' - 'done_when 3, verbatim: "Measured at authoring and recorded, each with a negative control that fails: every mapped llama.cpp `binary` exists in the pinned build (d1d3c3396) and every mapped flag appears in its `--help`; every ollama `subcommand` and flag appears in `ollama --help` (0.33.2); every mapped apr flag is an arg of that verb in the surface." MEASURED: 7/7 llama.cpp binaries and 10/10 flags ok on gx10''s pinned build; 7/7 ollama subcommands plus `create --quantize` ok on lambda''s 0.33.2; every mapped apr flag is present in its verb''s surface args. The controls `llama-cli --frobnicate`, bin `llama-frobnicate` and `ollama frobnicate` all FAIL.' - 'done_when 4, out of scope, verbatim: "the `hf` and `llamafile` mappings are declared here but verified against their pins by the engine worker''s rows, since the pins don''t exist yet; the SHACL reading is aprender-97''s S2; the cells for newly mapped verbs are later #3739 slices."' + - 'Added to #3774 at ~19:30Z (the cop: "You own the correspondence file columns (`hf` and `llamafile`) and the judge side (#3774)"), verbatim: "`scripts/lib/crux_inference_judge.py` treats `hf` and `llamafile` as comparators: RED when either is right and apr is wrong." MEASURED by `bash scripts/check_crux_inference_judge.sh`: 33 ok, 0 broke. New rows cover: hf right with apr wrong is RED; a llamafile refusal is quoted and, beside a right llama.cpp, the cell is GREEN; hf output that is not the contract JSON, or JSON with no `text`, is no oracle.' + - 'Verbatim: "`tok` rows ... and `tmpl` rows ... are byte-equal or RED, with the first differing index or byte named. An engine row with no apr row beside it is RED. A row the reference engine refused is UNJUDGED, and the run declines. `greedy` rows are REPORTED." MEASURED rows: equal tok ids are GREEN; differing ids are RED at index 1; an HF tok row without apr is RED; an HF-refused tok row declines rc 2; an equal tmpl is GREEN; the #3672 double wrap is RED at byte 17; greedy divergence at step 2 is reported with its logit cosine 0.7071 and no verdict.' + - 'Verbatim: "`scripts/crux_inference_dogfood.sh` exports `CRUX_MANIFEST` and `CRUX_WORK`, resolves `scripts/crux_engine_.{sh,py}` ... A missing script, a failing probe, or a `gen` that appends no row becomes a refused row naming why." MEASURED on lambda with a throwaway fake hf driver (never committed): probe recorded verbatim as `hf.probe`; 3 hf gen rows judged correct; the absent llamafile driver refused by name on every cell; the receipt table has 5 engine columns. The driver still reads no clock (0 rate tokens) and check_no_competing_harnesses stays count=0.' + - 'Mutation for the judge side: dropping the tok/tmpl mismatch-or-absent RED breaks 3 rows; not failing the run on a deterministic RED breaks 3; not declining a deterministic UNJUDGED breaks 1; dropping hf as a comparator breaks 1; not requiring the contract `text` field breaks 1; taking the logit cosine at the wrong step breaks 1. The slice-1 mutants still hold.' phases: [] subtasks: [] estimated_effort: null labels: - kind:code - notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' + notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml, scripts/lib/crux_inference_judge.py, scripts/crux_inference_dogfood.sh (the plugin-engine hook), scripts/check_crux_inference_judge.sh, and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' diff --git a/evidence/crux/verb-correspondence.yaml b/evidence/crux/verb-correspondence.yaml index 83ee461750..8b51181a71 100644 --- a/evidence/crux/verb-correspondence.yaml +++ b/evidence/crux/verb-correspondence.yaml @@ -4,8 +4,8 @@ # 19:03Z / 19:04Z amendments of #3739. Engine pins are the engine worker's rows. schema: crux-verb-correspondence/v1 surface: - source: 'apr surface --json (apr-cli-surface/v1, #3745 S1), emitted by apr 0.69.0 (621a47862)' - sha256: a7c6b7f1b66aaae231363904456bd0a6b13d98dfdfb5c9064b5f7407076e6886 + source: 'apr surface --json (apr-cli-surface/v1.1, #3745 S1), emitted by apr 0.69.0 (7ed033b63)' + sha256: 207c329715dc08211676b81e6ae6d61915393924902798747f9a0c5e7b36da2b leaves: 253 engines: - llama.cpp @@ -401,9 +401,7 @@ verbs: llamafile: binary: llamafile args: - --system: -sys - --max-tokens: -n - --temperature: --temp + : --server - key: check comparators: llama.cpp: @@ -438,7 +436,8 @@ verbs: args: {} llamafile: binary: llamafile - args: {} + args: + : --server - key: compare-hf comparators: llama.cpp: @@ -1240,13 +1239,13 @@ verbs: - key: mcp comparators: llama.cpp: - none: apr's MCP server; no engine ships an MCP server + none: apr's MCP server; no engine ships an MCP server. Its generation runs through the engine the `run` and `serve run` cells already compare ollama: - none: apr's MCP server; no engine ships an MCP server + none: apr's MCP server; no engine ships an MCP server. Its generation runs through the engine the `run` and `serve run` cells already compare hf: - none: apr's MCP server; no engine ships an MCP server + none: apr's MCP server; no engine ships an MCP server. Its generation runs through the engine the `run` and `serve run` cells already compare llamafile: - none: apr's MCP server; no engine ships an MCP server + none: apr's MCP server; no engine ships an MCP server. Its generation runs through the engine the `run` and `serve run` cells already compare - key: merge comparators: llama.cpp: @@ -1352,13 +1351,13 @@ verbs: - key: parity comparators: llama.cpp: - none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + none: compares apr's OWN GPU and CPU generations (apr vs apr); the external comparison of the same prompts on the same file is the `run` cells ollama: - none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + none: compares apr's OWN GPU and CPU generations (apr vs apr); the external comparison of the same prompts on the same file is the `run` cells hf: - none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + none: compares apr's OWN GPU and CPU generations (apr vs apr); the external comparison of the same prompts on the same file is the `run` cells llamafile: - none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + none: compares apr's OWN GPU and CPU generations (apr vs apr); the external comparison of the same prompts on the same file is the `run` cells - key: pipeline apply comparators: llama.cpp: @@ -2121,13 +2120,13 @@ verbs: - key: rosetta compare-inference comparators: llama.cpp: - none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + none: compares apr's inference across formats of one model (apr vs apr); the cross-engine comparison of the same file is the `run` cells ollama: - none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + none: compares apr's inference across formats of one model (apr vs apr); the cross-engine comparison of the same file is the `run` cells hf: - none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + none: compares apr's inference across formats of one model (apr vs apr); the cross-engine comparison of the same file is the `run` cells llamafile: - none: an apr-internal integrity, diagnostic or packaging verb; no inference engine has an equivalent + none: compares apr's inference across formats of one model (apr vs apr); the cross-engine comparison of the same file is the `run` cells - key: rosetta convert comparators: llama.cpp: @@ -2209,10 +2208,7 @@ verbs: llamafile: binary: llamafile args: - --prompt: -p - --max-tokens: -n - --temperature: --temp - --seed: --seed + : --server - key: runs diff comparators: llama.cpp: @@ -2271,6 +2267,7 @@ verbs: llamafile: binary: llamafile args: + : --server --port: --port --context-length: -c - key: shard diff --git a/scripts/check_crux_inference_judge.sh b/scripts/check_crux_inference_judge.sh index 816a74b8e2..af3b7eddbe 100755 --- a/scripts/check_crux_inference_judge.sh +++ b/scripts/check_crux_inference_judge.sh @@ -78,8 +78,55 @@ tokrow() { # tokrow # The manifest rows carry the literal "%s" for the sha; fill it in one place. seal() { sed -i "s/\"%s\"/\"$SHA\"/g" "$1"; } +engine_out() { # engine_out — row contract v1: {"text": ...} + python3 -c 'import json,sys; json.dump({"text": sys.argv[2], "reported": {"reported_by": "fixture"}}, open(sys.argv[1], "w"))' \ + "$1/$2-$3.json" "$4" +} +detrow() { # detrow [thinking] [refused] + python3 - "$@" <<'PY' +import json, sys +m, kind, eng, pid, path = sys.argv[1:6] +thinking = sys.argv[6] if len(sys.argv) > 6 else "unset" +ref = sys.argv[7] if len(sys.argv) > 7 else "" +row = {"kind": kind, "engine": eng, "model_sha256": "%s", "host": "fixture", "prompt_id": pid, "refused": ref or None} +if kind == "tok": + row.update({"input": path + ".txt", "ids": path}) +elif kind == "tmpl": + row.update({"thinking": thinking, "messages": path + ".messages.json", "rendered": path}) +else: + row.update({"steps": 4, "tokens": path, "logits": path + ".npy"}) +open(m, "a").write(json.dumps(row) + "\n") +PY +} +npy() { # npy — a float32 C-order 2-D .npy, stdlib only + python3 - "$1" "$2" <<'PY' +import struct, sys +rows = [[float(x) for x in r.split(",")] for r in sys.argv[2].split(";")] +n, m = len(rows), len(rows[0]) +hdr = ("{'descr': ': the positive control answered right by apr and llama.cpp + apr_out "$1" golden-2plus2 "4" gpu false; llama_out "$1" golden-2plus2 "What is 2+2?" "4" + row "$1/manifest.jsonl" apr golden-2plus2 0 "$1/apr-golden-2plus2.out" "$1/apr-golden-2plus2.err" + row "$1/manifest.jsonl" llama.cpp golden-2plus2 0 "$1/llama-golden-2plus2.out" "$1/llama-golden-2plus2.err" +} +det_verdict() { # det_verdict -> " " + python3 -c 'import json,sys +r = json.load(open(sys.argv[1])) +d = next((x for x in r.get("deterministic", []) if x["kind"] == sys.argv[2] and x["key"]["prompt_id"] == sys.argv[3]), None) +if d is None: print("ABSENT"); sys.exit(0) +ref = next(iter(d["references"].values()), {}) +print(d["verdict"], ref.get("first_difference"))' "$1/receipt.json" "$2" "$3" 2>/dev/null +} + run_judge() { # run_judge — prints the receipt path; returns the judge's rc - printf '{"version": "0.0.0", "host": "fixture", "backend": "gpu", "engines": ["apr", "llama.cpp", "ollama"]}\n' > "$1/meta.json" + printf '{"version": "0.0.0", "host": "fixture", "backend": "gpu", "engines": %s}\n' \ + "${META_ENGINES:-[\"apr\", \"llama.cpp\", \"ollama\"]}" > "$1/meta.json" seal "$1/manifest.jsonl" python3 "$JUDGE" collect --manifest "$1/manifest.jsonl" --prompts "$PROMPTS" --meta "$1/meta.json" \ --out-json "$1/receipt.json" --out-md "$1/receipt.md" > "$1/judge.out" 2> "$1/judge.err" @@ -265,6 +312,88 @@ row "$d/manifest.jsonl" llama.cpp golden-paris 0 "$d/llama-golden-paris.out" "$d run_judge "$d"; GOT_RC=$? expect "a model with no measured control cell declines" "$d" 2 golden-paris GREEN +# ---- the 19:03Z / 19:04Z engines: hf and llamafile (row contract v1) -------------- +META_ENGINES='["apr", "llama.cpp", "ollama", "hf", "llamafile"]' + +# E1. hf right, apr wrong: RED, exactly as for llama.cpp or ollama. +d=$(newcase hf_right_apr_wrong) +apr_out "$d" $P "5" gpu false; engine_out "$d" hf $P "2 + 2 = 4" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" hf $P 0 "$d/hf-$P.json" "" +run_judge "$d"; GOT_RC=$? +expect "hf right and apr wrong is RED" "$d" 1 $P RED + +# E2. llamafile refuses the model (its own text quoted); llama.cpp answers: still judged, refusal named. +d=$(newcase llamafile_refused) +apr_out "$d" $P "4" gpu false; llama_out "$d" $P "$Q" "4" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" llama.cpp $P 0 "$d/llama-$P.out" "$d/llama-$P.err" +row "$d/manifest.jsonl" llamafile $P "" "" "" "error loading model architecture: unknown model architecture: 'qwen35'" +row "$d/manifest.jsonl" hf $P 0 "$d/hf-$P.json" ""; engine_out "$d" hf $P "4" +run_judge "$d"; GOT_RC=$? +expect "a llamafile refusal beside a right llama.cpp is GREEN" "$d" 0 $P GREEN +why=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["cells"][0]["engines"]["llamafile"]["why"])' "$d/receipt.json" 2>/dev/null) +case "$why" in *"unknown model architecture: 'qwen35'"*) ok "llamafile's refusal is quoted, not dropped" ;; *) broke "llamafile refusal: '$why'" ;; esac + +# E3. an hf answer that is not the contract's JSON is no answer, so it cannot vouch. +d=$(newcase hf_bad_json) +apr_out "$d" $P "5" gpu false; printf '2 + 2 = 4\n' > "$d/hf-$P.json" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" hf $P 0 "$d/hf-$P.json" "" +run_judge "$d"; GOT_RC=$? +expect "hf output that is not the contract JSON is no oracle (UNJUDGED)" "$d" 2 $P UNJUDGED + +# E3b. valid JSON that is not the contract's shape (no "text") is no answer either. +d=$(newcase hf_json_without_text) +apr_out "$d" $P "5" gpu false; printf '{"answer": "4"}\n' > "$d/hf-$P.json" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" hf $P 0 "$d/hf-$P.json" "" +run_judge "$d"; GOT_RC=$? +expect "hf JSON without a text field is no oracle (UNJUDGED)" "$d" 2 $P UNJUDGED + +# D1-D4. tok: raw-text ids, BYTE-EQUAL or RED. +d=$(newcase tok_equal); control_green "$d" +printf '{"tokens": [9707, 11, 1879]}\n' > "$d/apr-tok.json"; printf '{"tokens": [9707, 11, 1879]}\n' > "$d/hf-tok.json" +detrow "$d/manifest.jsonl" tok apr tok-cjk-01 "$d/apr-tok.json"; detrow "$d/manifest.jsonl" tok hf tok-cjk-01 "$d/hf-tok.json" +run_judge "$d"; GOT_RC=$?; got=$(det_verdict "$d" tok tok-cjk-01) +[ "$GOT_RC" = 0 ] && [ "$got" = "GREEN None" ] && ok "tok ids equal to HF are GREEN (rc 0)" || broke "tok equal: rc $GOT_RC, '$got'" +d=$(newcase tok_differ); control_green "$d" +printf '{"tokens": [9707, 0, 0, 0]}\n' > "$d/apr-tok.json"; printf '{"tokens": [9707, 11, 1879]}\n' > "$d/hf-tok.json" +detrow "$d/manifest.jsonl" tok apr tok-cjk-01 "$d/apr-tok.json"; detrow "$d/manifest.jsonl" tok hf tok-cjk-01 "$d/hf-tok.json" +run_judge "$d"; GOT_RC=$?; got=$(det_verdict "$d" tok tok-cjk-01) +[ "$GOT_RC" = 1 ] && [ "$got" = "RED 1" ] && ok "tok ids differing from HF are RED at the first differing index (1)" || broke "tok differ: rc $GOT_RC, '$got'" +d=$(newcase tok_apr_missing); control_green "$d" +printf '{"tokens": [9707]}\n' > "$d/hf-tok.json"; detrow "$d/manifest.jsonl" tok hf tok-cjk-01 "$d/hf-tok.json" +run_judge "$d"; GOT_RC=$?; got=$(det_verdict "$d" tok tok-cjk-01) +[ "$GOT_RC" = 1 ] && [ "${got%% *}" = RED ] && ok "an HF tok row with no apr row is RED (absence)" || broke "tok apr missing: rc $GOT_RC, '$got'" +d=$(newcase tok_hf_refused); control_green "$d" +printf '{"tokens": [9707]}\n' > "$d/apr-tok.json"; detrow "$d/manifest.jsonl" tok apr tok-cjk-01 "$d/apr-tok.json" +detrow "$d/manifest.jsonl" tok hf tok-cjk-01 "$d/hf-tok.json" unset "no tokenizer.json in the source repo" +run_judge "$d"; GOT_RC=$?; got=$(det_verdict "$d" tok tok-cjk-01) +[ "$GOT_RC" = 2 ] && [ "${got%% *}" = UNJUDGED ] && ok "a tok row HF refused is UNJUDGED and declines" || broke "tok hf refused: rc $GOT_RC, '$got'" + +# D5-D6. tmpl: the chat rendering, BYTE-EQUAL or RED (the #3672 double wrap is the RED case). +d=$(newcase tmpl_equal); control_green "$d" +printf '<|im_start|>user\nhi<|im_end|>\n<|im_start|>assistant\n' > "$d/apr-t"; cp "$d/apr-t" "$d/hf-t" +detrow "$d/manifest.jsonl" tmpl apr golden-2plus2 "$d/apr-t" off; detrow "$d/manifest.jsonl" tmpl hf golden-2plus2 "$d/hf-t" off +run_judge "$d"; GOT_RC=$?; got=$(det_verdict "$d" tmpl golden-2plus2) +[ "$GOT_RC" = 0 ] && [ "$got" = "GREEN None" ] && ok "a rendering byte-equal to apply_chat_template is GREEN" || broke "tmpl equal: rc $GOT_RC, '$got'" +d=$(newcase tmpl_double_wrap); control_green "$d" +printf '<|im_start|>user\n<\xe2\x80\x8b|im_start|>user\nhi' > "$d/apr-t"; printf '<|im_start|>user\nhi<|im_end|>\n' > "$d/hf-t" +detrow "$d/manifest.jsonl" tmpl apr golden-2plus2 "$d/apr-t" off; detrow "$d/manifest.jsonl" tmpl hf golden-2plus2 "$d/hf-t" off +run_judge "$d"; GOT_RC=$?; got=$(det_verdict "$d" tmpl golden-2plus2) +[ "$GOT_RC" = 1 ] && [ "$got" = "RED 17" ] && ok "a double-wrapped rendering is RED at byte 17, where the second wrap starts" || broke "tmpl differ: rc $GOT_RC, '$got'" + +# R1. greedy: the first divergence and the logit cosine there are REPORTED, never a verdict. +d=$(newcase greedy_report); control_green "$d" +printf '{"prompt_ids": [1], "generated_ids": [5, 6, 7, 8]}\n' > "$d/apr-g.json"; npy "$d/apr-g.json.npy" "1,0;0,1;1,1;0,0" +printf '{"prompt_ids": [1], "generated_ids": [5, 6, 9, 8]}\n' > "$d/hf-g.json"; npy "$d/hf-g.json.npy" "1,0;0,1;1,0;0,0" +detrow "$d/manifest.jsonl" greedy apr golden-paris "$d/apr-g.json"; detrow "$d/manifest.jsonl" greedy hf golden-paris "$d/hf-g.json" +run_judge "$d"; GOT_RC=$? +got=$(python3 -c 'import json,sys; g=json.load(open(sys.argv[1]))["greedy"][0]["hf"]; print(g.get("first_divergence"), round(g.get("logit_cosine_at_divergence") or -9, 4))' "$d/receipt.json" 2>/dev/null) +[ "$GOT_RC" = 0 ] && [ "$got" = "2 0.7071" ] && ok "greedy divergence at step 2 is reported with its logit cosine (0.7071), not judged" || broke "greedy: rc $GOT_RC, '$got'" +unset META_ENGINES + # 10-11. token parity: equal ids agree; a divergence is located, never averaged away. d=$(newcase parity) apr_out "$d" $P "4" gpu false "151644,872,198,3838"; llama_out "$d" $P "$Q" "4" diff --git a/scripts/crux_inference_dogfood.sh b/scripts/crux_inference_dogfood.sh index 24f60f0d76..9419b732cb 100755 --- a/scripts/crux_inference_dogfood.sh +++ b/scripts/crux_inference_dogfood.sh @@ -10,7 +10,7 @@ # ollama for same model: chat/serve/run/code, etc". # # Usage: bash scripts/crux_inference_dogfood.sh --model [--model ]... -# [--host ] [--backend gpu|cpu] [--engines apr,llama.cpp,ollama] +# [--host ] [--backend gpu|cpu] [--engines apr,llama.cpp,ollama,hf,llamafile] # [--verbs run] [--out ] [--prompts ] [--timeout ] # [--keep-ollama-models] [--keep-work] # the apr version under test; `apr --version` must report it, or @@ -37,6 +37,12 @@ # llama scripts/llama_bin.sh ONLY ($LLAMA_BENCH_PATH names the pinned build). # ollama $OLLAMA_BIN, else the forjar-declared $HOME/.local/bin/ollama, else # /usr/local/bin/ollama; the SERVER's /api/version is what is recorded. +# hf, llamafile (the 19:03Z / 19:04Z amendments) are PLUGIN engines: the +# engine worker's scripts/crux_engine_.{sh,py} (row contract v1, +# #3739 issuecomment-5765991210) with `probe` and `gen` subcommands. It +# appends its own rows to $CRUX_MANIFEST. An absent script, a failing +# probe or a gen that appends no row is a REFUSED row naming why, never +# an absence. # sampling temperature, seed and context come from scripts/llama_pin.toml # [protocol], the declaration the parity gates already read. # @@ -62,7 +68,7 @@ decline() { printf 'decline: %s\n' "$*" >&2; exit 2; } VERSION="" HOST_ID="" BACKEND="gpu" -ENGINES="apr,llama.cpp,ollama" +ENGINES="apr,llama.cpp,ollama,hf,llamafile" VERBS="run" OUT_DIR="" PROMPTS="scripts/crux_inference_prompts.json" @@ -166,7 +172,33 @@ if want ollama; then fi fi fi -[ "$LLAMA_OK" = 1 ] || [ "$OLLAMA_OK" = 1 ] || decline "no comparator: llama.cpp (${LLAMA_WHY:-not requested}); ollama (${OLLAMA_WHY:-not requested})" +# ---- plugin engines (hf, llamafile): resolved by script, recorded by their probe -- +declare -A EXT_OK EXT_WHY EXT_SCRIPT EXT_PROBE +for eng in hf llamafile; do + want "$eng" || continue + EXT_OK[$eng]=0 + for f in "scripts/crux_engine_$eng.sh" "scripts/crux_engine_$eng.py"; do + [ -f "$f" ] && { EXT_SCRIPT[$eng]="$f"; break; } + done + if [ -z "${EXT_SCRIPT[$eng]:-}" ]; then + EXT_WHY[$eng]="engine driver scripts/crux_engine_$eng.{sh,py} is not present (the engine worker's row, #3739)" + continue + fi + case "${EXT_SCRIPT[$eng]}" in *.py) ext_run=(python3) ;; *) ext_run=(bash) ;; esac + probe_out=$("${ext_run[@]}" "${EXT_SCRIPT[$eng]}" probe 2> /tmp/crux-probe-$$-$eng.err) + prc=$? + if [ "$prc" -eq 0 ] && [ -n "$probe_out" ]; then + EXT_OK[$eng]=1 + EXT_PROBE[$eng]=$(printf '%s\n' "$probe_out" | head -1) + else + EXT_WHY[$eng]="probe exit $prc: $(head -c 300 /tmp/crux-probe-$$-$eng.err 2>/dev/null | tr '\n' ' ')" + fi + rm -f /tmp/crux-probe-$$-$eng.err +done +EXT_ANY=0 +for eng in "${!EXT_OK[@]}"; do [ "${EXT_OK[$eng]}" = 1 ] && EXT_ANY=1; done + +[ "$LLAMA_OK" = 1 ] || [ "$OLLAMA_OK" = 1 ] || [ "$EXT_ANY" = 1 ] || decline "no comparator: llama.cpp (${LLAMA_WHY:-not requested}); ollama (${OLLAMA_WHY:-not requested})" # ---- work dir ------------------------------------------------------------------- WORK=$(mktemp -d) || decline "mktemp failed" @@ -188,6 +220,8 @@ _cleanup() { } trap _cleanup EXIT MANIFEST="$WORK/manifest.jsonl"; : > "$MANIFEST" +# The plugin engines append their own rows here (row contract v1). +export CRUX_MANIFEST="$MANIFEST" CRUX_WORK="$WORK" MODELS_JSONL="$WORK/models.jsonl"; : > "$MODELS_JSONL" # One file per prompt: its id list, and each prompt's last user message. @@ -255,6 +289,19 @@ open(m, "a").write(json.dumps(row) + "\n") PY } +rows_for() { # rows_for : how many gen rows that engine has for the prompt + python3 - "$MANIFEST" "$1" "$2" "$SHA" 9>&- <<'PY' +import json, sys +m, eng, pid, sha = sys.argv[1:5] +n = 0 +for line in open(m): + if line.strip(): + r = json.loads(line) + n += r.get("kind") == "gen" and r.get("engine") == eng and r.get("prompt_id") == pid and r.get("model_sha256") == sha +print(n) +PY +} + free_port() { python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])'; } # llama.cpp's OWN rendering and tokenization of each prompt: the reference apr's @@ -416,6 +463,29 @@ PY elif want ollama; then emit_gen ollama "$pid" "" "" "" "${OL_REFUSED:-$OLLAMA_WHY}" fi + + for eng in hf llamafile; do + want "$eng" || continue + if [ "${EXT_OK[$eng]}" != 1 ]; then + emit_gen "$eng" "$pid" "" "" "" "${EXT_WHY[$eng]}" + continue + fi + case "${EXT_SCRIPT[$eng]}" in *.py) ext_run=(python3) ;; *) ext_run=(bash) ;; esac + # llamafile's `--cli` ignores the thinking switch and does not bound output + # with -n (infra-3c, measured on 0.10.6); its server honours both. + ext_extra=() + [ "$eng" = llamafile ] && ext_extra=(--interface server) + before=$(rows_for "$eng" "$pid") + eng_run "$d/$eng-$pid.driver.out" "$d/$eng-$pid.driver.err" "${ext_run[@]}" "${EXT_SCRIPT[$eng]}" gen \ + --model "$M" --model-sha256 "$SHA" --verb "$VERB" --prompt-id "$pid" \ + --messages "$WORK/messages-$pid.json" --prompt-file "$WORK/prompt-$pid.txt" \ + --thinking "$THINK" --backend "$BACKEND" --host "$HOST" \ + --max-tokens "$MAXTOK" --seed "$SEED" --temperature "$TEMP" --context "$CTX" "${ext_extra[@]}" + rc=$? + if [ "$(rows_for "$eng" "$pid")" -le "$before" ]; then + emit_gen "$eng" "$pid" "" "" "" "engine driver ${EXT_SCRIPT[$eng]} gen exited $rc without appending a row: $(tail -c 200 "$d/$eng-$pid.driver.err" 2>/dev/null | tr '\n' ' ')" + fi + done cell_unlock done @@ -427,12 +497,19 @@ done # ---- judge -------------------------------------------------------------------------- GPU_NAME=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | head -1) HARNESS_SHA=$(git -C "$ROOT" rev-parse --short HEAD 2>/dev/null) || HARNESS_SHA="" +EXT_META="$WORK/ext-engines.json" +python3 - "$EXT_META" "${EXT_PROBE[hf]:-}" "${EXT_WHY[hf]:-}" "${EXT_PROBE[llamafile]:-}" "${EXT_WHY[llamafile]:-}" <<'PY' +import json, sys +out, hp, hw, lp, lw = sys.argv[1:6] +json.dump({"hf": {"probe": hp or None, "unavailable": hw or None}, + "llamafile": {"probe": lp or None, "unavailable": lw or None}}, open(out, "w")) +PY python3 - "$WORK/meta.json" "$MODELS_JSONL" "$VERSION" "$HOST" "$BACKEND" "$ENGINES" "$VERBS" \ "$APR_VERSION_LINE" "${LLAMA_BUILD:-}" "$(llama_pin_get build_commit 2>/dev/null)" "$LLAMA_WHY" \ - "$OLLAMA_SERVER" "$OLLAMA_CLIENT" "$OLLAMA_WHY" "$TEMP" "$SEED" "$CTX" "$MAXTOK" "${GPU_NAME:-}" "$HARNESS_SHA" <<'PY' + "$OLLAMA_SERVER" "$OLLAMA_CLIENT" "$OLLAMA_WHY" "$TEMP" "$SEED" "$CTX" "$MAXTOK" "${GPU_NAME:-}" "$HARNESS_SHA" "$EXT_META" <<'PY' import json, platform, sys (out, models, version, host, backend, engines, verbs, apr_line, lbuild, lpin, lwhy, - osrv, ocli, owhy, temp, seed, ctx, maxtok, gpu, hsha) = sys.argv[1:21] + osrv, ocli, owhy, temp, seed, ctx, maxtok, gpu, hsha, ext) = sys.argv[1:22] meta = { "version": version, "host": host, "backend": backend, "isa": platform.machine(), "gpu": gpu or None, "engines": engines.split(","), "verbs": verbs.split(","), "thinking": ["off"], @@ -442,6 +519,7 @@ meta = { "harness": {"sha": hsha or None, "driver": "scripts/crux_inference_dogfood.sh"}, "llama_cpp": {"build": lbuild or None, "pin": lpin or None, "unavailable": lwhy or None}, "ollama": {"server_version": osrv or None, "client_version": ocli or None, "unavailable": owhy or None}, + **json.load(open(ext)), "models": [json.loads(l) for l in open(models) if l.strip()], "not_covered": [ "verbs chat, serve and code (the next slices of #3739)", diff --git a/scripts/lib/crux_inference_judge.py b/scripts/lib/crux_inference_judge.py index d65462a48d..9aa620ba06 100644 --- a/scripts/lib/crux_inference_judge.py +++ b/scripts/lib/crux_inference_judge.py @@ -45,13 +45,16 @@ """ import argparse +import ast import datetime import json +import math import re +import struct import sys -ENGINES = ("apr", "llama.cpp", "ollama") -COMPARATORS = ("llama.cpp", "ollama") +COMPARATORS = ("llama.cpp", "ollama", "hf", "llamafile") +ENGINES = ("apr",) + COMPARATORS ANSI = re.compile(r"\x1b\[[0-9;?]*[A-Za-z]|\x1b\][^\x07]*\x07|\r") @@ -172,6 +175,24 @@ def num(key): return out +def parse_engine_json(stdout): + """hf and llamafile rows (row contract v1, #3739 issuecomment-5765991210): + the engine driver writes `{"text": , "reported": {...}}`.""" + out = {"answer": None, "why": None, "reported": {}} + try: + doc = json.loads(stdout) + except ValueError as exc: + out["why"] = "stdout is not the contract's JSON: %s" % exc + return out + if not isinstance(doc, dict) or not isinstance(doc.get("text"), str): + out["why"] = "stdout JSON has no text field" + return out + rep = doc.get("reported") + out["reported"] = rep if isinstance(rep, dict) else {} + out["answer"] = doc["text"] + return out + + # ------------------------------------------------------------------ judge -- @@ -202,6 +223,10 @@ def engine_entry(row, prompt): p = parse_llamacpp_cli(stdout, content) elif engine == "ollama": p = parse_ollama(stdout, stderr) + elif engine in ("hf", "llamafile"): + p = parse_engine_json(stdout) + if row.get("source"): + e["source"] = row["source"] else: e["why"] = "unknown engine %r" % engine return e @@ -251,6 +276,138 @@ def token_parity(apr_entry, tok_row): } +# ------------------------------------------- deterministic rows (row contract v1) -- +# tok and tmpl are BYTE-EQUAL or RED (#3739, 19:03Z): an engine that produced +# ids / a rendering and an apr that produced a different one, or none, is RED. +# greedy is REPORTED: the judge holds no tokenizer, so it cannot decide from ids +# alone whether a divergence changed the answer; that verdict stays with `gen`. + + +def _load_json(path, field): + with open(path, encoding="utf-8") as fh: + v = json.load(fh).get(field) + if not isinstance(v, list): + raise ValueError("%s has no list %r" % (path, field)) + return v + + +def _load_bytes(path): + with open(path, "rb") as fh: + return fh.read() + + +def _first_diff(a, b): + i = next((i for i, (x, y) in enumerate(zip(a, b)) if x != y), None) + if i is None and len(a) != len(b): + i = min(len(a), len(b)) + return i + + +def _det_side(row, loader, field): + if row is None: + return None, "missing: no row for this engine" + if row.get("refused"): + return None, "refused: " + row["refused"] + try: + return (loader(row[field]) if field else None), None + except (OSError, ValueError, KeyError, TypeError) as exc: + return None, "unreadable: %s" % exc + + +def judge_deterministic(rows, kind): + """kind 'tok': rows carrying `input` (raw-text ids). kind 'tmpl': chat renderings.""" + if kind == "tok": + rows = [r for r in rows if r.get("kind") == "tok" and "input" in r] + keyf = lambda r: (r["model_sha256"], r["host"], r["prompt_id"]) + names = ("model_sha256", "host", "prompt_id") + load = lambda r: _det_side(r, lambda p: _load_json(p, "tokens"), "ids") + else: + rows = [r for r in rows if r.get("kind") == "tmpl"] + keyf = lambda r: (r["model_sha256"], r["host"], r["prompt_id"], r.get("thinking", "unset")) + names = ("model_sha256", "host", "prompt_id", "thinking") + load = lambda r: _det_side(r, _load_bytes, "rendered") + groups = {} + for r in rows: + groups.setdefault(keyf(r), {})[r["engine"]] = r + out = [] + for key in sorted(groups): + by = groups[key] + apr_val, apr_why = load(by.get("apr")) + refs = {} + for eng, r in sorted(by.items()): + if eng == "apr": + continue + val, why = load(r) + refs[eng] = {"produced": val is not None, "why": why} + if val is not None and apr_val is not None: + d = _first_diff(apr_val, val) + refs[eng]["equal"] = d is None + refs[eng]["first_difference"] = d + produced = [e for e, v in refs.items() if v["produced"]] + if not produced: + verdict = "UNJUDGED" + elif apr_val is None or any(not refs[e]["equal"] for e in produced): + verdict = "RED" + else: + verdict = "GREEN" + out.append({"kind": kind, "key": dict(zip(names, key)), "verdict": verdict, + "apr": {"produced": apr_val is not None, "why": apr_why}, "references": refs}) + return out + + +def _npy_rows(path): + """A float32 C-order 2-D .npy, read with the stdlib (runners have no numpy).""" + data = _load_bytes(path) + if data[:6] != b"\x93NUMPY": + raise ValueError("not an .npy file") + major = data[6] + hlen = struct.unpack(" Date: Mon, 21 Sep 2026 23:13:40 +0200 Subject: [PATCH 3/5] PMAT-3774: cells run as ONE gpu-q command (the cop's rule rev 5), HF source weights declared by GGUF sha at pinned revisions, and a degenerate answer is no answer (the integration run's '!' x64 was scored correct by the golden '!' pattern, #3782) Refs #3774 Co-Authored-By: Claude Opus 5 (1M context) --- docs/roadmaps/entries/PMAT-3774.yaml | 8 +- docs/roadmaps/roadmap.yaml | 8 +- evidence/crux/hf-sources.yaml | 18 +++ scripts/check_crux_inference_judge.sh | 14 ++ scripts/crux_inference_dogfood.sh | 202 +++++++++++++++----------- scripts/lib/crux_inference_judge.py | 17 +++ 6 files changed, 180 insertions(+), 87 deletions(-) create mode 100644 evidence/crux/hf-sources.yaml diff --git a/docs/roadmaps/entries/PMAT-3774.yaml b/docs/roadmaps/entries/PMAT-3774.yaml index 3f71400578..28ce0eac8f 100644 --- a/docs/roadmaps/entries/PMAT-3774.yaml +++ b/docs/roadmaps/entries/PMAT-3774.yaml @@ -6,7 +6,7 @@ priority: critical assigned_to: aprender-76 created: 2026-09-21T19:20:00Z - updated: 2026-09-21T21:05:00Z + updated: 2026-09-21T21:20:00Z spec: null acceptance_criteria: - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the `apr-cli-surface/v1.1` surface emitted by apr 0.69.0 (7ed033b63, branch PMAT-3745-s1-surface-emitter), sha256 207c3297…da2b. The entry key set equals the leaf key set, with no duplicates. Every `generates: true` leaf that maps no engine (embed, rerank, mcp, parity, rosetta compare-inference) has a specific reason; parity and compare-inference compare apr with apr, and their external comparison is the `run` cells.' @@ -17,9 +17,13 @@ - 'Verbatim: "`tok` rows ... and `tmpl` rows ... are byte-equal or RED, with the first differing index or byte named. An engine row with no apr row beside it is RED. A row the reference engine refused is UNJUDGED, and the run declines. `greedy` rows are REPORTED." MEASURED rows: equal tok ids are GREEN; differing ids are RED at index 1; an HF tok row without apr is RED; an HF-refused tok row declines rc 2; an equal tmpl is GREEN; the #3672 double wrap is RED at byte 17; greedy divergence at step 2 is reported with its logit cosine 0.7071 and no verdict.' - 'Verbatim: "`scripts/crux_inference_dogfood.sh` exports `CRUX_MANIFEST` and `CRUX_WORK`, resolves `scripts/crux_engine_.{sh,py}` ... A missing script, a failing probe, or a `gen` that appends no row becomes a refused row naming why." MEASURED on lambda with a throwaway fake hf driver (never committed): probe recorded verbatim as `hf.probe`; 3 hf gen rows judged correct; the absent llamafile driver refused by name on every cell; the receipt table has 5 engine columns. The driver still reads no clock (0 rate tokens) and check_no_competing_harnesses stays count=0.' - 'Mutation for the judge side: dropping the tok/tmpl mismatch-or-absent RED breaks 3 rows; not failing the run on a deterministic RED breaks 3; not declining a deterministic UNJUDGED breaks 1; dropping hf as a comparator breaks 1; not requiring the contract `text` field breaks 1; taking the logit cosine at the wrong step breaks 1. The slice-1 mutants still hold.' + - 'Added ~21:15Z, verbatim from #3774: "The GPU lock goes through gpu-q ... Each cell (one prompt, every engine) is written as one script and runs as ONE gpu-q command." MEASURED: the driver has no fd-9 lock left (0 hits for 9>&-/cell_lock/eng_run). On the gpu lane it runs `GPUQ_WAIT=$LOCK_WAIT gpu-q --prio ${CRUX_GPU_PRIO:-5} -- bash ` when `gpu-q --caps` lists `wait` (it does on lambda and gx10), else the bounded flock; `harness.gpu_lock` records which. On the cpu lane it records "none (cpu lane, the rule''s clause 2)".' + - 'Verbatim: "`evidence/crux/hf-sources.yaml` maps each GGUF (by sha256) to the HF source weights at a PINNED revision." MEASURED: cc324af0… maps to Qwen/Qwen2.5-Coder-1.5B-Instruct@2e1fd397 (bfloat16). Its GGUF provenance was checked through the HF API: the LFS sha256 of qwen2.5-coder-1.5b-instruct-q4_k_m.gguf in Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF@f86cb2c1 equals the local file''s.' + - 'Verbatim: "A degenerate answer is no answer, from any engine." MEASURED: case rows cover apr''s `!!!!…` as RED where llama.cpp answered, and hf''s `!!!!…` as unable to vouch. The mutant that removes the rule breaks 2 rows. The table is 35 ok, 0 broke.' + - 'INTEGRATION, lambda cpu lane, with infra-3c''s REAL engine drivers (PMAT-3778 @ e719fc11a, copied in untracked for the run and removed after): the hf probe was recorded verbatim (transformers 5.17.0, torch 2.14.0+cu130, lock sha256 be29a4f4…); hf gen rows were appended and judged; the absent llamafile install was refused by name ("probe exit 3: llamafile is not installed at … (forjar resource crux-llamafile, paiml/infra#906)"). It FOUND that the hf engine answered `!` × 64 on all 3 prompts (transformers left lm_head untied; reported to infra-3c), and that the golden greeting pattern scored it correct: #3782 and the degenerate rule above.' phases: [] subtasks: [] estimated_effort: null labels: - kind:code - notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml, scripts/lib/crux_inference_judge.py, scripts/crux_inference_dogfood.sh (the plugin-engine hook), scripts/check_crux_inference_judge.sh, and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' + notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml, evidence/crux/hf-sources.yaml, scripts/lib/crux_inference_judge.py, scripts/crux_inference_dogfood.sh (the plugin-engine hook), scripts/check_crux_inference_judge.sh, and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 579e61158d..91b58978da 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -19999,7 +19999,7 @@ roadmap: priority: critical assigned_to: aprender-76 created: 2026-09-21T19:20:00Z - updated: 2026-09-21T21:05:00Z + updated: 2026-09-21T21:20:00Z spec: null acceptance_criteria: - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the `apr-cli-surface/v1.1` surface emitted by apr 0.69.0 (7ed033b63, branch PMAT-3745-s1-surface-emitter), sha256 207c3297…da2b. The entry key set equals the leaf key set, with no duplicates. Every `generates: true` leaf that maps no engine (embed, rerank, mcp, parity, rosetta compare-inference) has a specific reason; parity and compare-inference compare apr with apr, and their external comparison is the `run` cells.' @@ -20010,9 +20010,13 @@ roadmap: - 'Verbatim: "`tok` rows ... and `tmpl` rows ... are byte-equal or RED, with the first differing index or byte named. An engine row with no apr row beside it is RED. A row the reference engine refused is UNJUDGED, and the run declines. `greedy` rows are REPORTED." MEASURED rows: equal tok ids are GREEN; differing ids are RED at index 1; an HF tok row without apr is RED; an HF-refused tok row declines rc 2; an equal tmpl is GREEN; the #3672 double wrap is RED at byte 17; greedy divergence at step 2 is reported with its logit cosine 0.7071 and no verdict.' - 'Verbatim: "`scripts/crux_inference_dogfood.sh` exports `CRUX_MANIFEST` and `CRUX_WORK`, resolves `scripts/crux_engine_.{sh,py}` ... A missing script, a failing probe, or a `gen` that appends no row becomes a refused row naming why." MEASURED on lambda with a throwaway fake hf driver (never committed): probe recorded verbatim as `hf.probe`; 3 hf gen rows judged correct; the absent llamafile driver refused by name on every cell; the receipt table has 5 engine columns. The driver still reads no clock (0 rate tokens) and check_no_competing_harnesses stays count=0.' - 'Mutation for the judge side: dropping the tok/tmpl mismatch-or-absent RED breaks 3 rows; not failing the run on a deterministic RED breaks 3; not declining a deterministic UNJUDGED breaks 1; dropping hf as a comparator breaks 1; not requiring the contract `text` field breaks 1; taking the logit cosine at the wrong step breaks 1. The slice-1 mutants still hold.' + - 'Added ~21:15Z, verbatim from #3774: "The GPU lock goes through gpu-q ... Each cell (one prompt, every engine) is written as one script and runs as ONE gpu-q command." MEASURED: the driver has no fd-9 lock left (0 hits for 9>&-/cell_lock/eng_run). On the gpu lane it runs `GPUQ_WAIT=$LOCK_WAIT gpu-q --prio ${CRUX_GPU_PRIO:-5} -- bash ` when `gpu-q --caps` lists `wait` (it does on lambda and gx10), else the bounded flock; `harness.gpu_lock` records which. On the cpu lane it records "none (cpu lane, the rule''s clause 2)".' + - 'Verbatim: "`evidence/crux/hf-sources.yaml` maps each GGUF (by sha256) to the HF source weights at a PINNED revision." MEASURED: cc324af0… maps to Qwen/Qwen2.5-Coder-1.5B-Instruct@2e1fd397 (bfloat16). Its GGUF provenance was checked through the HF API: the LFS sha256 of qwen2.5-coder-1.5b-instruct-q4_k_m.gguf in Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF@f86cb2c1 equals the local file''s.' + - 'Verbatim: "A degenerate answer is no answer, from any engine." MEASURED: case rows cover apr''s `!!!!…` as RED where llama.cpp answered, and hf''s `!!!!…` as unable to vouch. The mutant that removes the rule breaks 2 rows. The table is 35 ok, 0 broke.' + - 'INTEGRATION, lambda cpu lane, with infra-3c''s REAL engine drivers (PMAT-3778 @ e719fc11a, copied in untracked for the run and removed after): the hf probe was recorded verbatim (transformers 5.17.0, torch 2.14.0+cu130, lock sha256 be29a4f4…); hf gen rows were appended and judged; the absent llamafile install was refused by name ("probe exit 3: llamafile is not installed at … (forjar resource crux-llamafile, paiml/infra#906)"). It FOUND that the hf engine answered `!` × 64 on all 3 prompts (transformers left lm_head untied; reported to infra-3c), and that the golden greeting pattern scored it correct: #3782 and the degenerate rule above.' phases: [] subtasks: [] estimated_effort: null labels: - kind:code - notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml, scripts/lib/crux_inference_judge.py, scripts/crux_inference_dogfood.sh (the plugin-engine hook), scripts/check_crux_inference_judge.sh, and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' + notes: 'Refs #3774 (child of #3739; the closing keyword goes on this PR only if the quorum passes it). SCOPE: evidence/crux/verb-correspondence.yaml, evidence/crux/hf-sources.yaml, scripts/lib/crux_inference_judge.py, scripts/crux_inference_dogfood.sh (the plugin-engine hook), scripts/check_crux_inference_judge.sh, and this fragment. STACKED on crux-3739-inference-dogfood (PR #3767), so the diff to judge is `origin/crux-3739-inference-dogfood...HEAD`. HOW IT IS MADE: a one-off authoring script, kept out of the tree, reads `apr surface --json` and writes the YAML. The committed file is the declaration, bound to its surface by sha256; the header says to re-derive when that sha changes. Engine-level `none` reasons state a fact about the engine. The mapped set follows the cop''s examples in the 17:19Z amendment (serve ↔ llama-server / ollama serve, tokenize ↔ llama-tokenize, bench ↔ llama-bench, quantize ↔ llama-quantize, pull/list ↔ ollama pull/list) plus the verbs whose counterpart is a real engine verb on the same file. `finetune` is `none`: at the pin, llama-finetune''s `--lora` loads an adapter and does not train one. `tokenize train/plan/apply/import-hf/repair-manifest` are tokenizer training/format verbs and map to none; only `encode-corpus` (text → ids) maps, to llama-tokenize / AutoTokenizer. Each mapped engine becomes a REQUIRED :CruxCell once S2 reads this file, so a mapping is a commitment, not a label.' diff --git a/evidence/crux/hf-sources.yaml b/evidence/crux/hf-sources.yaml new file mode 100644 index 0000000000..ba4fc8dd16 --- /dev/null +++ b/evidence/crux/hf-sources.yaml @@ -0,0 +1,18 @@ +# evidence/crux/hf-sources.yaml (#3774, the hf engine of #3739). +# The HF column runs the SOURCE weights a GGUF was quantized from (the 19:03Z +# amendment: "GGUF Q4_K cells ... HF runs the source bf16 weights"). This maps each +# GGUF, by its sha256, to that source at a PINNED revision; a moving branch is not +# a pin. The GGUF side is provenance: its repo's LFS sha256 must equal the file's. +# A GGUF with no entry here is a refused hf row naming the gap, never an absence. +schema: crux-hf-sources/v1 +sources: + cc324af070c2ecbfd324a30884d2f951a7ff756aba85cb811a6ec436933bb046: + gguf: + repo: Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF + revision: f86cb2c1fa58255f8052cc32aeede1b7482d4361 + file: qwen2.5-coder-1.5b-instruct-q4_k_m.gguf + lfs_sha256_matches: true # measured 2026-09-21 via the HF API (?blobs=true) + hf: + repo: Qwen/Qwen2.5-Coder-1.5B-Instruct + revision: 2e1fd397ee46e1388853d2af2c993145b0f1098a + dtype: bfloat16 diff --git a/scripts/check_crux_inference_judge.sh b/scripts/check_crux_inference_judge.sh index af3b7eddbe..5ff33cea07 100755 --- a/scripts/check_crux_inference_judge.sh +++ b/scripts/check_crux_inference_judge.sh @@ -351,6 +351,20 @@ row "$d/manifest.jsonl" hf $P 0 "$d/hf-$P.json" "" run_judge "$d"; GOT_RC=$? expect "hf JSON without a text field is no oracle (UNJUDGED)" "$d" 2 $P UNJUDGED +# E4. a DEGENERATE answer is no answer from any engine: the golden greeting's "!" +# pattern would otherwise score "!!!!!!!!" (token id 0, x64) as correct. +d=$(newcase degenerate_answers) +apr_out "$d" golden-greeting "!!!!!!!!!!!!!!!!" gpu false; engine_out "$d" hf golden-greeting "!!!!!!!!!!!!!!!!" +llama_out "$d" golden-greeting "Hello there, how are you doing today my friend?" "Hello! I am well." +row "$d/manifest.jsonl" apr golden-greeting 0 "$d/apr-golden-greeting.out" "$d/apr-golden-greeting.err" +row "$d/manifest.jsonl" hf golden-greeting 0 "$d/hf-golden-greeting.json" "" +row "$d/manifest.jsonl" llama.cpp golden-greeting 0 "$d/llama-golden-greeting.out" "$d/llama-golden-greeting.err" +control_green "$d" +run_judge "$d"; GOT_RC=$? +expect "apr's degenerate '!!!!' is no answer: RED where llama.cpp answered" "$d" 1 golden-greeting RED +got=$(python3 -c 'import json,sys; c=[x for x in json.load(open(sys.argv[1]))["cells"] if x["key"]["prompt_id"]=="golden-greeting"][0]; print(c["engines"]["hf"]["answered"], c["engines"]["hf"]["why"][:18])' "$d/receipt.json" 2>/dev/null) +case "$got" in "False degenerate output"*) ok "hf's degenerate '!!!!' cannot vouch either" ;; *) broke "hf degenerate: '$got'" ;; esac + # D1-D4. tok: raw-text ids, BYTE-EQUAL or RED. d=$(newcase tok_equal); control_green "$d" printf '{"tokens": [9707, 11, 1879]}\n' > "$d/apr-tok.json"; printf '{"tokens": [9707, 11, 1879]}\n' > "$d/hf-tok.json" diff --git a/scripts/crux_inference_dogfood.sh b/scripts/crux_inference_dogfood.sh index 9419b732cb..66913b9303 100755 --- a/scripts/crux_inference_dogfood.sh +++ b/scripts/crux_inference_dogfood.sh @@ -51,11 +51,12 @@ # receipt, labelled by engine, and judges none of them. Nothing in this file # reads a clock. A throughput CLAIM goes through scripts/perf_gate.sh (PERF-009). # -# GPU sharing (the cop's rule, /mnt/nvme-raid0/agent-wt/gpu-lock-rule.txt): -# on the gpu lane each cell holds /tmp/apr-gpu.lock for its engine invocations -# only, and every engine runs under `choom -n 1000`, so a measurement is the OOM -# victim, never a CI runner. llama.cpp and ollama are held to it as well as apr, -# and ollama runs with keep_alive 0 and must leave VRAM before the lock drops. +# GPU sharing (the cop's rule, /mnt/nvme-raid0/agent-wt/gpu-lock-rule.txt, rev 5): +# on the gpu lane each cell (one prompt, every engine) runs as ONE command through +# gpu-q (priority queue in front of /tmp/apr-gpu.lock; CRUX_GPU_PRIO, default 5), +# falling back to a bounded flock recorded as "may have jumped the queue". Every +# engine runs under `choom -n 1000`, so a measurement is the OOM victim, never a +# CI runner, and ollama must leave VRAM before the cell ends. # # Every status is captured as `cmd; rc=$?`, never through a pipe (#2336, #2360). set -uo pipefail @@ -72,6 +73,7 @@ ENGINES="apr,llama.cpp,ollama,hf,llamafile" VERBS="run" OUT_DIR="" PROMPTS="scripts/crux_inference_prompts.json" +HF_SOURCES="${CRUX_HF_SOURCES:-evidence/crux/hf-sources.yaml}" TMO=600 LOCK_WAIT=3600 KEEP_OLLAMA=0 @@ -195,6 +197,15 @@ for eng in hf llamafile; do fi rm -f /tmp/crux-probe-$$-$eng.err done +GPUQ="${GPUQ_BIN:-$HOME/.local/bin/gpu-q}" +GPUQ_OK=0 +GPU_PRIO="${CRUX_GPU_PRIO:-5}" +LOCK_VIA="flock /tmp/apr-gpu.lock (no gpu-q with \`wait\` on this host: may have jumped the queue)" +if [ -x "$GPUQ" ]; then + gpuq_caps=$("$GPUQ" --caps 2>/dev/null) + case "$gpuq_caps" in *wait*) GPUQ_OK=1; LOCK_VIA="gpu-q --prio $GPU_PRIO (GPUQ_WAIT bound)" ;; esac +fi +[ "$BACKEND" = gpu ] || LOCK_VIA="none (cpu lane, the rule's clause 2)" EXT_ANY=0 for eng in "${!EXT_OK[@]}"; do [ "${EXT_OK[$eng]}" = 1 ] && EXT_ANY=1; done @@ -236,47 +247,58 @@ PY ) || decline "prompt set $PROMPTS unreadable" MAXTOK=$(python3 -c 'import json,sys; print(int(json.load(open(sys.argv[1]))["max_tokens"]))' "$PROMPTS") || decline "max_tokens unreadable" -# THE CELL LOCK (the cop's GPU rule rev 2, plus its #3739 ruling on ollama). On -# the gpu lane one cell, meaning one prompt through every engine, holds -# /tmp/apr-gpu.lock for exactly its engine invocations. It releases the lock only -# after ollama's model has left VRAM: ollama keeps a model resident (keep_alive, -# default 5 min), and a resident model would starve the next session's cell on a -# 24 GB card. flock exits 75 (not 1) when the lock is not had in time, so a lock -# wait is never read as an engine's own failure. The cpu lane touches no GPU -# compute and takes no lock (the rule's clause 2); every engine still runs under -# choom, so a measurement is the OOM victim, never a CI runner. -cell_lock() { - [ "$BACKEND" = gpu ] || return 0 - exec 9> /tmp/apr-gpu.lock || return 75 - flock -w "$LOCK_WAIT" -E 75 9 +# ONE CELL = ONE COMMAND UNDER THE GPU LOCK (the cop's rule rev 5, #3739). A cell is +# one prompt through every engine. Its engine commands are written into one +# script and that script runs as ONE command: `GPUQ_WAIT= gpu-q --prio P +# -- bash ` when this host's gpu-q advertises `wait`, else the bounded +# `flock -w ... -E 75 /tmp/apr-gpu.lock choom -n 1000 -- bash `, recorded +# as "may have jumped the queue". gpu-q serves (priority, arrival) in front of +# the same flock; a bare flock is not FIFO and starved a P0 head for 113 min. +# ollama runs with keep_alive 0 and the cell waits for `ollama ps` to drop the +# model BEFORE the script exits, so VRAM is free when the lock is released. The +# cpu lane touches no GPU compute and runs the same script with no lock (the +# rule's clause 2), under choom. Exit 75 from either tool means the lock was not +# had within the bound: every engine of that cell is a refused row, never skipped. +cell_add() { # cell_add cmd... — one engine line: out, err, rc files + local cell="$1" prefix="$2"; shift 2 + { printf 'timeout %q ' "$TMO"; printf '%q ' "$@" + printf '> %q 2> %q < /dev/null; echo $? > %q\n' "$prefix.out" "$prefix.err" "$prefix.rc"; } >> "$cell" } -cell_unlock() { - [ "$BACKEND" = gpu ] || return 0 - flock -u 9 - exec 9>&- +cell_add_ollama_unload() { # cell_add_ollama_unload + { printf '%q stop %q > /dev/null 2>&1; u=false\n' "$OLLAMA" "$3" + printf 'for i in $(seq 1 30); do l=$(%q ps 2> /dev/null); case "$l" in *%q*) sleep 1 ;; *) u=true; break ;; esac; done\n' "$OLLAMA" "$3" + printf 'echo "$u" > %q\n' "$2.unloaded"; } >> "$1" } - -# eng_run cmd... — one engine invocation. `9>&-` keeps the lock -# fd out of the engine, so nothing an engine leaves running can hold the lock. -eng_run() { - local o="$1" e="$2"; shift 2 - choom -n 1000 -- timeout "$TMO" "$@" > "$o" 2> "$e" < /dev/null 9>&- +run_cell() { # run_cell ; sets CELL_WHY when the lock was not had + local rc + CELL_WHY="" + if [ "$BACKEND" != gpu ]; then + choom -n 1000 -- bash "$1" + return $? + fi + if [ "$GPUQ_OK" = 1 ]; then + GPUQ_WAIT="$LOCK_WAIT" "$GPUQ" --prio "$GPU_PRIO" -- bash "$1" 2> "$1.lock.err" + rc=$? + else + flock -w "$LOCK_WAIT" -E 75 /tmp/apr-gpu.lock choom -n 1000 -- bash "$1" 2> "$1.lock.err" + rc=$? + fi + if [ "$rc" -ne 0 ]; then + CELL_WHY="the GPU lock was not had ($LOCK_VIA, rc $rc, bound ${LOCK_WAIT}s): $(tail -c 200 "$1.lock.err" 2>/dev/null | tr '\n' ' ')" + fi + return "$rc" } - -# ollama_unloaded : stop the model, then wait up to 30 s for `ollama ps` to -# stop listing it. Prints true or false, and the cell records which. -ollama_unloaded() { - local i listed - "$OLLAMA" stop "$1" > /dev/null 2>&1 9>&- - for i in $(seq 1 30); do - listed=$("$OLLAMA" ps 2>/dev/null 9>&-) - case "$listed" in *"$1"*) sleep 1 ;; *) printf 'true'; return 0 ;; esac - done - printf 'false' +cell_result() { # cell_result — the engine's row from its cell files + local rc unl="" + if [ -n "$CELL_WHY" ]; then emit_gen "$1" "$2" "" "" "" "$CELL_WHY"; return; fi + if [ ! -f "$3.rc" ]; then emit_gen "$1" "$2" "" "" "" "the cell ran but this engine's line never finished"; return; fi + rc=$(cat "$3.rc") + [ -f "$3.unloaded" ] && unl=$(cat "$3.unloaded") + emit_gen "$1" "$2" "$rc" "$3.out" "$3.err" "" "$unl" } emit_gen() { # emit_gen [ollama_unloaded] - python3 - "$MANIFEST" "$1" "$2" "$3" "$4" "$5" "$6" "${7:-}" "$SHA" "$HOST" "$VERB" "$THINK" "$BACKEND" 9>&- <<'PY' + python3 - "$MANIFEST" "$1" "$2" "$3" "$4" "$5" "$6" "${7:-}" "$SHA" "$HOST" "$VERB" "$THINK" "$BACKEND" <<'PY' import json, sys m, eng, pid, rc, o, e, ref, unl, sha, host, verb, think, be = sys.argv[1:14] row = {"kind": "gen", "engine": eng, "prompt_id": pid, @@ -290,7 +312,7 @@ PY } rows_for() { # rows_for : how many gen rows that engine has for the prompt - python3 - "$MANIFEST" "$1" "$2" "$SHA" 9>&- <<'PY' + python3 - "$MANIFEST" "$1" "$2" "$SHA" <<'PY' import json, sys m, eng, pid, sha = sys.argv[1:5] n = 0 @@ -367,6 +389,29 @@ for M_IN in "${MODELS[@]}"; do mkdir -p "$WORK/$SHA12" printf ' model %s sha256 %s\n' "$NAME" "$SHA" + # hf runs the SOURCE weights of this GGUF, declared by sha256 in + # evidence/crux/hf-sources.yaml at a pinned revision; no entry = a refused hf row. + HF_SRC=() + HF_MODEL_WHY="" + if want hf && [ "${EXT_OK[hf]:-0}" = 1 ]; then + hf_line=$(python3 - "$HF_SOURCES" "$SHA" <<'PY' +import sys, yaml +try: + src = (yaml.safe_load(open(sys.argv[1])) or {}).get("sources", {}).get(sys.argv[2], {}).get("hf") or {} +except OSError: + src = {} +if src.get("repo") and src.get("revision"): + print("%s\t%s\t%s" % (src["repo"], src["revision"], src.get("dtype", "bfloat16"))) +PY +) + if [ -n "$hf_line" ]; then + IFS=$'\t' read -r hf_repo hf_rev hf_dtype <<< "$hf_line" + HF_SRC=(--source-repo "$hf_repo" --source-revision "$hf_rev" --dtype "$hf_dtype") + else + HF_MODEL_WHY="no HF source declared for this GGUF (sha256 ${SHA:0:12}…) in $HF_SOURCES" + fi + fi + THINKING_CAPABLE=unknown TOK_WHY="" if [ "$LLAMA_OK" = 1 ]; then @@ -431,62 +476,53 @@ PY content=$(cat "$WORK/prompt-$pid.txt") d="$WORK/$SHA12/$VERB" mkdir -p "$d" + cell="$d/cell-$pid.sh" + printf '#!/usr/bin/env bash\n# one CRUX cell: prompt %s through every engine, one hold of the GPU lock\n' "$pid" > "$cell" - if ! cell_lock; then - ref="the GPU lock was not had within ${LOCK_WAIT}s" - emit_gen apr "$pid" "" "" "" "$ref" - want llama.cpp && emit_gen llama.cpp "$pid" "" "" "" "$ref" - want ollama && emit_gen ollama "$pid" "" "" "" "$ref" - continue - fi - - eng_run "$d/apr-$pid.out" "$d/apr-$pid.err" "$APR" run "$M" --prompt "$content" --max-tokens "$MAXTOK" \ + cell_add "$cell" "$d/apr-$pid" "$APR" run "$M" --prompt "$content" --max-tokens "$MAXTOK" \ --temperature "$TEMP" --seed "$SEED" --format json -v "$APR_BE" - rc=$? - emit_gen apr "$pid" "$rc" "$d/apr-$pid.out" "$d/apr-$pid.err" "" - - if [ "$LLAMA_OK" = 1 ]; then - eng_run "$d/llama-$pid.out" "$d/llama-$pid.err" "$LLAMA_CLI" -m "$M" -p "$content" -st -n "$MAXTOK" \ - --temp "$TEMP" --seed "$SEED" -c "$CTX" -ngl "$NGL" "${LLAMA_DEV[@]}" "${LLAMA_THINK[@]}" - rc=$? - emit_gen llama.cpp "$pid" "$rc" "$d/llama-$pid.out" "$d/llama-$pid.err" "" - elif want llama.cpp; then - emit_gen llama.cpp "$pid" "" "" "" "$LLAMA_WHY" - fi - + [ "$LLAMA_OK" = 1 ] && cell_add "$cell" "$d/llama-$pid" "$LLAMA_CLI" -m "$M" -p "$content" -st -n "$MAXTOK" \ + --temp "$TEMP" --seed "$SEED" -c "$CTX" -ngl "$NGL" "${LLAMA_DEV[@]}" "${LLAMA_THINK[@]}" if [ "$OLLAMA_OK" = 1 ] && [ -z "$OL_REFUSED" ]; then - eng_run "$d/ollama-$pid.out" "$d/ollama-$pid.err" "$OLLAMA" run "$OL_NAME" "$content" --verbose --nowordwrap \ + cell_add "$cell" "$d/ollama-$pid" "$OLLAMA" run "$OL_NAME" "$content" --verbose --nowordwrap \ --keepalive 0 "${OLLAMA_THINK[@]}" - rc=$? - unl=$(ollama_unloaded "$OL_NAME") - emit_gen ollama "$pid" "$rc" "$d/ollama-$pid.out" "$d/ollama-$pid.err" "" "$unl" - elif want ollama; then - emit_gen ollama "$pid" "" "" "" "${OL_REFUSED:-$OLLAMA_WHY}" + cell_add_ollama_unload "$cell" "$d/ollama-$pid" "$OL_NAME" fi - for eng in hf llamafile; do - want "$eng" || continue - if [ "${EXT_OK[$eng]}" != 1 ]; then - emit_gen "$eng" "$pid" "" "" "" "${EXT_WHY[$eng]}" - continue - fi + want "$eng" && [ "${EXT_OK[$eng]}" = 1 ] || continue + [ "$eng" = hf ] && [ -n "$HF_MODEL_WHY" ] && continue case "${EXT_SCRIPT[$eng]}" in *.py) ext_run=(python3) ;; *) ext_run=(bash) ;; esac # llamafile's `--cli` ignores the thinking switch and does not bound output # with -n (infra-3c, measured on 0.10.6); its server honours both. ext_extra=() [ "$eng" = llamafile ] && ext_extra=(--interface server) - before=$(rows_for "$eng" "$pid") - eng_run "$d/$eng-$pid.driver.out" "$d/$eng-$pid.driver.err" "${ext_run[@]}" "${EXT_SCRIPT[$eng]}" gen \ + [ "$eng" = hf ] && ext_extra=("${HF_SRC[@]}") + cell_add "$cell" "$d/$eng-$pid.driver" "${ext_run[@]}" "${EXT_SCRIPT[$eng]}" gen \ --model "$M" --model-sha256 "$SHA" --verb "$VERB" --prompt-id "$pid" \ --messages "$WORK/messages-$pid.json" --prompt-file "$WORK/prompt-$pid.txt" \ --thinking "$THINK" --backend "$BACKEND" --host "$HOST" \ --max-tokens "$MAXTOK" --seed "$SEED" --temperature "$TEMP" --context "$CTX" "${ext_extra[@]}" - rc=$? - if [ "$(rows_for "$eng" "$pid")" -le "$before" ]; then - emit_gen "$eng" "$pid" "" "" "" "engine driver ${EXT_SCRIPT[$eng]} gen exited $rc without appending a row: $(tail -c 200 "$d/$eng-$pid.driver.err" 2>/dev/null | tr '\n' ' ')" + done + printf 'exit 0\n' >> "$cell" + + before_hf=$(rows_for hf "$pid"); before_lf=$(rows_for llamafile "$pid") + run_cell "$cell" + + cell_result apr "$pid" "$d/apr-$pid" + if [ "$LLAMA_OK" = 1 ]; then cell_result llama.cpp "$pid" "$d/llama-$pid" + elif want llama.cpp; then emit_gen llama.cpp "$pid" "" "" "" "$LLAMA_WHY"; fi + if [ "$OLLAMA_OK" = 1 ] && [ -z "$OL_REFUSED" ]; then cell_result ollama "$pid" "$d/ollama-$pid" + elif want ollama; then emit_gen ollama "$pid" "" "" "" "${OL_REFUSED:-$OLLAMA_WHY}"; fi + for eng in hf llamafile; do + want "$eng" || continue + if [ "${EXT_OK[$eng]}" != 1 ]; then emit_gen "$eng" "$pid" "" "" "" "${EXT_WHY[$eng]}"; continue; fi + if [ "$eng" = hf ] && [ -n "$HF_MODEL_WHY" ]; then emit_gen hf "$pid" "" "" "" "$HF_MODEL_WHY"; continue; fi + before=$before_hf; [ "$eng" = llamafile ] && before=$before_lf + if [ -n "$CELL_WHY" ]; then emit_gen "$eng" "$pid" "" "" "" "$CELL_WHY" + elif [ "$(rows_for "$eng" "$pid")" -le "$before" ]; then + emit_gen "$eng" "$pid" "" "" "" "engine driver ${EXT_SCRIPT[$eng]} gen exited $(cat "$d/$eng-$pid.driver.rc" 2>/dev/null || echo '?') without appending a row: $(tail -c 200 "$d/$eng-$pid.driver.err" 2>/dev/null | tr '\n' ' ')" fi done - cell_unlock done if [ "$OLLAMA_OK" = 1 ] && [ "$KEEP_OLLAMA" = 0 ]; then @@ -506,17 +542,17 @@ json.dump({"hf": {"probe": hp or None, "unavailable": hw or None}, PY python3 - "$WORK/meta.json" "$MODELS_JSONL" "$VERSION" "$HOST" "$BACKEND" "$ENGINES" "$VERBS" \ "$APR_VERSION_LINE" "${LLAMA_BUILD:-}" "$(llama_pin_get build_commit 2>/dev/null)" "$LLAMA_WHY" \ - "$OLLAMA_SERVER" "$OLLAMA_CLIENT" "$OLLAMA_WHY" "$TEMP" "$SEED" "$CTX" "$MAXTOK" "${GPU_NAME:-}" "$HARNESS_SHA" "$EXT_META" <<'PY' + "$OLLAMA_SERVER" "$OLLAMA_CLIENT" "$OLLAMA_WHY" "$TEMP" "$SEED" "$CTX" "$MAXTOK" "${GPU_NAME:-}" "$HARNESS_SHA" "$EXT_META" "$LOCK_VIA" <<'PY' import json, platform, sys (out, models, version, host, backend, engines, verbs, apr_line, lbuild, lpin, lwhy, - osrv, ocli, owhy, temp, seed, ctx, maxtok, gpu, hsha, ext) = sys.argv[1:22] + osrv, ocli, owhy, temp, seed, ctx, maxtok, gpu, hsha, ext, lockvia) = sys.argv[1:23] meta = { "version": version, "host": host, "backend": backend, "isa": platform.machine(), "gpu": gpu or None, "engines": engines.split(","), "verbs": verbs.split(","), "thinking": ["off"], "sampling": {"temperature": float(temp), "seed": int(seed), "context": int(ctx), "max_tokens": int(maxtok), "source": "scripts/llama_pin.toml [protocol]"}, "apr": {"version_line": apr_line}, - "harness": {"sha": hsha or None, "driver": "scripts/crux_inference_dogfood.sh"}, + "harness": {"sha": hsha or None, "driver": "scripts/crux_inference_dogfood.sh", "gpu_lock": lockvia}, "llama_cpp": {"build": lbuild or None, "pin": lpin or None, "unavailable": lwhy or None}, "ollama": {"server_version": osrv or None, "client_version": ocli or None, "unavailable": owhy or None}, **json.load(open(ext)), diff --git a/scripts/lib/crux_inference_judge.py b/scripts/lib/crux_inference_judge.py index 9aa620ba06..54b6d94388 100644 --- a/scripts/lib/crux_inference_judge.py +++ b/scripts/lib/crux_inference_judge.py @@ -196,6 +196,20 @@ def parse_engine_json(stdout): # ------------------------------------------------------------------ judge -- +def degenerate(text): + """A degenerate answer is NO answer, from any engine. Measured on #3774's + integration run: a broken HF load emitted token id 0 ("!") 64 times, and + the golden greeting's own pattern "!" (golden_output.rs) scored that CORRECT. + apr's #3726 failure mode is the same shape. Threshold: at least 8 non-space + characters, 90% or more of them one character, which no real answer to the + golden prompts ("4", "The capital of France is Paris.") comes near.""" + chars = [ch for ch in (text or "") if not ch.isspace()] + if len(chars) < 8: + return False + top = max(chars.count(ch) for ch in set(chars)) + return top >= 0.9 * len(chars) + + def correct(entry, expect_any): return entry.get("answered", False) and any(p in (entry.get("answer") or "") for p in expect_any) @@ -238,6 +252,9 @@ def engine_entry(row, prompt): if p["answer"] is None: e["why"] = p["why"] return e + if degenerate(p["answer"]): + e["why"] = "degenerate output (one character is >=90%% of it): %r" % p["answer"][:24] + return e be = p.get("backend") if engine == "apr" else None if be and (be.get("fell_back") or (row.get("backend") and be.get("ran") != row.get("backend"))): e["why"] = "backend: asked %s, ran %s (fell_back=%s)" % (row.get("backend"), be.get("ran"), be.get("fell_back")) From a330f010446236c2fb47854aa5183c22fe3be986 Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 23:35:14 +0200 Subject: [PATCH 4/5] PMAT-3774: plugin engines held to their lane (reported.device required), positive_control per cell (#3715), bashrs-clean array assignment Refs #3774 Co-Authored-By: Claude Opus 5 (1M context) --- docs/roadmaps/entries/PMAT-3774.yaml | 5 +++-- docs/roadmaps/roadmap.yaml | 5 +++-- scripts/check_crux_inference_judge.sh | 25 ++++++++++++++++++++++--- scripts/crux_inference_dogfood.sh | 5 ++++- scripts/lib/crux_inference_judge.py | 15 +++++++++++++++ 5 files changed, 47 insertions(+), 8 deletions(-) diff --git a/docs/roadmaps/entries/PMAT-3774.yaml b/docs/roadmaps/entries/PMAT-3774.yaml index 28ce0eac8f..ef4215f384 100644 --- a/docs/roadmaps/entries/PMAT-3774.yaml +++ b/docs/roadmaps/entries/PMAT-3774.yaml @@ -6,7 +6,7 @@ priority: critical assigned_to: aprender-76 created: 2026-09-21T19:20:00Z - updated: 2026-09-21T21:20:00Z + updated: 2026-09-21T21:50:00Z spec: null acceptance_criteria: - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the `apr-cli-surface/v1.1` surface emitted by apr 0.69.0 (7ed033b63, branch PMAT-3745-s1-surface-emitter), sha256 207c3297…da2b. The entry key set equals the leaf key set, with no duplicates. Every `generates: true` leaf that maps no engine (embed, rerank, mcp, parity, rosetta compare-inference) has a specific reason; parity and compare-inference compare apr with apr, and their external comparison is the `run` cells.' @@ -19,7 +19,8 @@ - 'Mutation for the judge side: dropping the tok/tmpl mismatch-or-absent RED breaks 3 rows; not failing the run on a deterministic RED breaks 3; not declining a deterministic UNJUDGED breaks 1; dropping hf as a comparator breaks 1; not requiring the contract `text` field breaks 1; taking the logit cosine at the wrong step breaks 1. The slice-1 mutants still hold.' - 'Added ~21:15Z, verbatim from #3774: "The GPU lock goes through gpu-q ... Each cell (one prompt, every engine) is written as one script and runs as ONE gpu-q command." MEASURED: the driver has no fd-9 lock left (0 hits for 9>&-/cell_lock/eng_run). On the gpu lane it runs `GPUQ_WAIT=$LOCK_WAIT gpu-q --prio ${CRUX_GPU_PRIO:-5} -- bash ` when `gpu-q --caps` lists `wait` (it does on lambda and gx10), else the bounded flock; `harness.gpu_lock` records which. On the cpu lane it records "none (cpu lane, the rule''s clause 2)".' - 'Verbatim: "`evidence/crux/hf-sources.yaml` maps each GGUF (by sha256) to the HF source weights at a PINNED revision." MEASURED: cc324af0… maps to Qwen/Qwen2.5-Coder-1.5B-Instruct@2e1fd397 (bfloat16). Its GGUF provenance was checked through the HF API: the LFS sha256 of qwen2.5-coder-1.5b-instruct-q4_k_m.gguf in Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF@f86cb2c1 equals the local file''s.' - - 'Verbatim: "A degenerate answer is no answer, from any engine." MEASURED: case rows cover apr''s `!!!!…` as RED where llama.cpp answered, and hf''s `!!!!…` as unable to vouch. The mutant that removes the rule breaks 2 rows. The table is 35 ok, 0 broke.' + - 'Verbatim: "A degenerate answer is no answer, from any engine." MEASURED: case rows cover apr''s `!!!!…` as RED where llama.cpp answered, and hf''s `!!!!…` as unable to vouch. The mutant that removes the rule breaks 2 rows. The table is 35 ok, 0 broke.' + - 'Also measured (~21:45Z): a plugin engine is held to its lane as apr is. An hf/llamafile gen row must carry `reported.device` matching its lane (`cpu*` on the cpu lane, anything else on gpu), and a missing device is no answer. This came out of the integration run, where HF on the cpu lane reached the 4090 outside the GPU lock (infra-3c confirmed it and fixed their side in 9c806f008). Rows: hf on the CPU in the gpu lane, and hf with no device, are both UNJUDGED. Each mutant breaks 1. Cells also carry `positive_control` (aprender-97''s additive field for pv). The table is 38 ok, 0 broke.' - 'INTEGRATION, lambda cpu lane, with infra-3c''s REAL engine drivers (PMAT-3778 @ e719fc11a, copied in untracked for the run and removed after): the hf probe was recorded verbatim (transformers 5.17.0, torch 2.14.0+cu130, lock sha256 be29a4f4…); hf gen rows were appended and judged; the absent llamafile install was refused by name ("probe exit 3: llamafile is not installed at … (forjar resource crux-llamafile, paiml/infra#906)"). It FOUND that the hf engine answered `!` × 64 on all 3 prompts (transformers left lm_head untied; reported to infra-3c), and that the golden greeting pattern scored it correct: #3782 and the degenerate rule above.' phases: [] subtasks: [] diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 91b58978da..54d57f6e0e 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -19999,7 +19999,7 @@ roadmap: priority: critical assigned_to: aprender-76 created: 2026-09-21T19:20:00Z - updated: 2026-09-21T21:20:00Z + updated: 2026-09-21T21:50:00Z spec: null acceptance_criteria: - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the `apr-cli-surface/v1.1` surface emitted by apr 0.69.0 (7ed033b63, branch PMAT-3745-s1-surface-emitter), sha256 207c3297…da2b. The entry key set equals the leaf key set, with no duplicates. Every `generates: true` leaf that maps no engine (embed, rerank, mcp, parity, rosetta compare-inference) has a specific reason; parity and compare-inference compare apr with apr, and their external comparison is the `run` cells.' @@ -20012,7 +20012,8 @@ roadmap: - 'Mutation for the judge side: dropping the tok/tmpl mismatch-or-absent RED breaks 3 rows; not failing the run on a deterministic RED breaks 3; not declining a deterministic UNJUDGED breaks 1; dropping hf as a comparator breaks 1; not requiring the contract `text` field breaks 1; taking the logit cosine at the wrong step breaks 1. The slice-1 mutants still hold.' - 'Added ~21:15Z, verbatim from #3774: "The GPU lock goes through gpu-q ... Each cell (one prompt, every engine) is written as one script and runs as ONE gpu-q command." MEASURED: the driver has no fd-9 lock left (0 hits for 9>&-/cell_lock/eng_run). On the gpu lane it runs `GPUQ_WAIT=$LOCK_WAIT gpu-q --prio ${CRUX_GPU_PRIO:-5} -- bash ` when `gpu-q --caps` lists `wait` (it does on lambda and gx10), else the bounded flock; `harness.gpu_lock` records which. On the cpu lane it records "none (cpu lane, the rule''s clause 2)".' - 'Verbatim: "`evidence/crux/hf-sources.yaml` maps each GGUF (by sha256) to the HF source weights at a PINNED revision." MEASURED: cc324af0… maps to Qwen/Qwen2.5-Coder-1.5B-Instruct@2e1fd397 (bfloat16). Its GGUF provenance was checked through the HF API: the LFS sha256 of qwen2.5-coder-1.5b-instruct-q4_k_m.gguf in Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF@f86cb2c1 equals the local file''s.' - - 'Verbatim: "A degenerate answer is no answer, from any engine." MEASURED: case rows cover apr''s `!!!!…` as RED where llama.cpp answered, and hf''s `!!!!…` as unable to vouch. The mutant that removes the rule breaks 2 rows. The table is 35 ok, 0 broke.' + - 'Verbatim: "A degenerate answer is no answer, from any engine." MEASURED: case rows cover apr''s `!!!!…` as RED where llama.cpp answered, and hf''s `!!!!…` as unable to vouch. The mutant that removes the rule breaks 2 rows. The table is 35 ok, 0 broke.' + - 'Also measured (~21:45Z): a plugin engine is held to its lane as apr is. An hf/llamafile gen row must carry `reported.device` matching its lane (`cpu*` on the cpu lane, anything else on gpu), and a missing device is no answer. This came out of the integration run, where HF on the cpu lane reached the 4090 outside the GPU lock (infra-3c confirmed it and fixed their side in 9c806f008). Rows: hf on the CPU in the gpu lane, and hf with no device, are both UNJUDGED. Each mutant breaks 1. Cells also carry `positive_control` (aprender-97''s additive field for pv). The table is 38 ok, 0 broke.' - 'INTEGRATION, lambda cpu lane, with infra-3c''s REAL engine drivers (PMAT-3778 @ e719fc11a, copied in untracked for the run and removed after): the hf probe was recorded verbatim (transformers 5.17.0, torch 2.14.0+cu130, lock sha256 be29a4f4…); hf gen rows were appended and judged; the absent llamafile install was refused by name ("probe exit 3: llamafile is not installed at … (forjar resource crux-llamafile, paiml/infra#906)"). It FOUND that the hf engine answered `!` × 64 on all 3 prompts (transformers left lm_head untied; reported to infra-3c), and that the golden greeting pattern scored it correct: #3782 and the degenerate rule above.' phases: [] subtasks: [] diff --git a/scripts/check_crux_inference_judge.sh b/scripts/check_crux_inference_judge.sh index 5ff33cea07..17c67880b7 100755 --- a/scripts/check_crux_inference_judge.sh +++ b/scripts/check_crux_inference_judge.sh @@ -78,9 +78,9 @@ tokrow() { # tokrow # The manifest rows carry the literal "%s" for the sha; fill it in one place. seal() { sed -i "s/\"%s\"/\"$SHA\"/g" "$1"; } -engine_out() { # engine_out — row contract v1: {"text": ...} - python3 -c 'import json,sys; json.dump({"text": sys.argv[2], "reported": {"reported_by": "fixture"}}, open(sys.argv[1], "w"))' \ - "$1/$2-$3.json" "$4" +engine_out() { # engine_out [device] — row contract v1: {"text": ...} + python3 -c 'import json,sys; json.dump({"text": sys.argv[2], "reported": {"reported_by": "fixture", "device": sys.argv[3]}}, open(sys.argv[1], "w"))' \ + "$1/$2-$3.json" "$4" "${5-cuda:0 fixture}" } detrow() { # detrow [thinking] [refused] python3 - "$@" <<'PY' @@ -303,6 +303,10 @@ case "$why" in *) broke "undeclared control: declined_because was '$why'" ;; esac +# 9e2. every cell says whether it is the positive control (pv reads the flag, #3715). +got=$(python3 -c 'import json,sys; r=json.load(open(sys.argv[1])); print(sorted((c["key"]["prompt_id"], c["positive_control"]) for c in r["cells"]))' "$TMP/control_all_wrong/receipt.json" 2>/dev/null) +[ "$got" = "[('golden-2plus2', True), ('golden-paris', False)]" ] && ok "cells carry positive_control (true only on the control)" || broke "positive_control flags: '$got'" + # 9f. a model whose run never measured the control prompt declines, however green: # an unmeasured control proves nothing about the harness. d=$(newcase no_control_cell) @@ -351,6 +355,21 @@ row "$d/manifest.jsonl" hf $P 0 "$d/hf-$P.json" "" run_judge "$d"; GOT_RC=$? expect "hf JSON without a text field is no oracle (UNJUDGED)" "$d" 2 $P UNJUDGED +# E3c. a plugin engine is held to its lane: a gpu-lane row that ran on the CPU, +# or a row with no reported device, cannot vouch. +d=$(newcase hf_wrong_device) +apr_out "$d" $P "5" gpu false; engine_out "$d" hf $P "4" "cpu" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" hf $P 0 "$d/hf-$P.json" "" +run_judge "$d"; GOT_RC=$? +expect "hf on the CPU in the gpu lane is no oracle (UNJUDGED)" "$d" 2 $P UNJUDGED +d=$(newcase hf_no_device) +apr_out "$d" $P "5" gpu false; engine_out "$d" hf $P "4" "" +row "$d/manifest.jsonl" apr $P 0 "$d/apr-$P.out" "$d/apr-$P.err" +row "$d/manifest.jsonl" hf $P 0 "$d/hf-$P.json" "" +run_judge "$d"; GOT_RC=$? +expect "hf with no reported device is no oracle (UNJUDGED)" "$d" 2 $P UNJUDGED + # E4. a DEGENERATE answer is no answer from any engine: the golden greeting's "!" # pattern would otherwise score "!!!!!!!!" (token id 0, x64) as correct. d=$(newcase degenerate_answers) diff --git a/scripts/crux_inference_dogfood.sh b/scripts/crux_inference_dogfood.sh index 66913b9303..f053f568d0 100755 --- a/scripts/crux_inference_dogfood.sh +++ b/scripts/crux_inference_dogfood.sh @@ -180,7 +180,10 @@ for eng in hf llamafile; do want "$eng" || continue EXT_OK[$eng]=0 for f in "scripts/crux_engine_$eng.sh" "scripts/crux_engine_$eng.py"; do - [ -f "$f" ] && { EXT_SCRIPT[$eng]="$f"; break; } + if [ -f "$f" ]; then + EXT_SCRIPT[$eng]="$f" + break + fi done if [ -z "${EXT_SCRIPT[$eng]:-}" ]; then EXT_WHY[$eng]="engine driver scripts/crux_engine_$eng.{sh,py} is not present (the engine worker's row, #3739)" diff --git a/scripts/lib/crux_inference_judge.py b/scripts/lib/crux_inference_judge.py index 54b6d94388..e2aee299e2 100644 --- a/scripts/lib/crux_inference_judge.py +++ b/scripts/lib/crux_inference_judge.py @@ -259,6 +259,19 @@ def engine_entry(row, prompt): if be and (be.get("fell_back") or (row.get("backend") and be.get("ran") != row.get("backend"))): e["why"] = "backend: asked %s, ran %s (fell_back=%s)" % (row.get("backend"), be.get("ran"), be.get("fell_back")) return e + if engine in ("hf", "llamafile"): + # A plugin engine is held to the lane as apr is. The integration run's + # cpu lane got `!` x64 from an HF load that went to CUDA anyway (and + # outside the GPU lock). The device must be REPORTED: an unverifiable + # lane cannot vouch for or against apr. + dev = str((p.get("reported") or {}).get("device") or "") + lane = row.get("backend") + if not dev: + e["why"] = "no reported.device: the %s lane cannot be verified" % lane + return e + if (lane == "cpu") != dev.lower().startswith("cpu"): + e["why"] = "device %r is not the %s lane" % (dev, lane) + return e e["answered"] = True return e @@ -483,6 +496,8 @@ def collect(args): cells.append({ "key": dict(zip(("model_sha256", "host", "verb", "thinking", "rung", "prompt_id"), k)), "verdict": verdict, + # additive (aprender-97, #3715): pv reads the control by this flag, never by a prompt name + "positive_control": bool(prompt.get("control")), "expect_any": prompt["expect_any"], "engines": entries, "agreement": { From c20ea060f84f96a5b9d8d5dfea144dc4a6611a0d Mon Sep 17 00:00:00 2001 From: Noah Gift Date: Mon, 21 Sep 2026 23:38:24 +0200 Subject: [PATCH 5/5] PMAT-3774: CRUX_GPU_PRIO defaults to 3 (the cop's class for CRUX development runs; the train's T-1 run passes 1) Refs #3774 Co-Authored-By: Claude Opus 5 (1M context) --- docs/roadmaps/entries/PMAT-3774.yaml | 4 ++-- docs/roadmaps/roadmap.yaml | 4 ++-- scripts/crux_inference_dogfood.sh | 6 ++++-- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/roadmaps/entries/PMAT-3774.yaml b/docs/roadmaps/entries/PMAT-3774.yaml index ef4215f384..cbaf078511 100644 --- a/docs/roadmaps/entries/PMAT-3774.yaml +++ b/docs/roadmaps/entries/PMAT-3774.yaml @@ -6,7 +6,7 @@ priority: critical assigned_to: aprender-76 created: 2026-09-21T19:20:00Z - updated: 2026-09-21T21:50:00Z + updated: 2026-09-21T22:00:00Z spec: null acceptance_criteria: - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the `apr-cli-surface/v1.1` surface emitted by apr 0.69.0 (7ed033b63, branch PMAT-3745-s1-surface-emitter), sha256 207c3297…da2b. The entry key set equals the leaf key set, with no duplicates. Every `generates: true` leaf that maps no engine (embed, rerank, mcp, parity, rosetta compare-inference) has a specific reason; parity and compare-inference compare apr with apr, and their external comparison is the `run` cells.' @@ -17,7 +17,7 @@ - 'Verbatim: "`tok` rows ... and `tmpl` rows ... are byte-equal or RED, with the first differing index or byte named. An engine row with no apr row beside it is RED. A row the reference engine refused is UNJUDGED, and the run declines. `greedy` rows are REPORTED." MEASURED rows: equal tok ids are GREEN; differing ids are RED at index 1; an HF tok row without apr is RED; an HF-refused tok row declines rc 2; an equal tmpl is GREEN; the #3672 double wrap is RED at byte 17; greedy divergence at step 2 is reported with its logit cosine 0.7071 and no verdict.' - 'Verbatim: "`scripts/crux_inference_dogfood.sh` exports `CRUX_MANIFEST` and `CRUX_WORK`, resolves `scripts/crux_engine_.{sh,py}` ... A missing script, a failing probe, or a `gen` that appends no row becomes a refused row naming why." MEASURED on lambda with a throwaway fake hf driver (never committed): probe recorded verbatim as `hf.probe`; 3 hf gen rows judged correct; the absent llamafile driver refused by name on every cell; the receipt table has 5 engine columns. The driver still reads no clock (0 rate tokens) and check_no_competing_harnesses stays count=0.' - 'Mutation for the judge side: dropping the tok/tmpl mismatch-or-absent RED breaks 3 rows; not failing the run on a deterministic RED breaks 3; not declining a deterministic UNJUDGED breaks 1; dropping hf as a comparator breaks 1; not requiring the contract `text` field breaks 1; taking the logit cosine at the wrong step breaks 1. The slice-1 mutants still hold.' - - 'Added ~21:15Z, verbatim from #3774: "The GPU lock goes through gpu-q ... Each cell (one prompt, every engine) is written as one script and runs as ONE gpu-q command." MEASURED: the driver has no fd-9 lock left (0 hits for 9>&-/cell_lock/eng_run). On the gpu lane it runs `GPUQ_WAIT=$LOCK_WAIT gpu-q --prio ${CRUX_GPU_PRIO:-5} -- bash ` when `gpu-q --caps` lists `wait` (it does on lambda and gx10), else the bounded flock; `harness.gpu_lock` records which. On the cpu lane it records "none (cpu lane, the rule''s clause 2)".' + - 'Added ~21:15Z, verbatim from #3774: "The GPU lock goes through gpu-q ... Each cell (one prompt, every engine) is written as one script and runs as ONE gpu-q command." MEASURED: the driver has no fd-9 lock left (0 hits for 9>&-/cell_lock/eng_run). On the gpu lane it runs `GPUQ_WAIT=$LOCK_WAIT gpu-q --prio ${CRUX_GPU_PRIO:-3} -- bash ` (the cop''s priority classes: CRUX development = 3, the train''s T-1 run passes 1) when `gpu-q --caps` lists `wait` (it does on lambda and gx10), else the bounded flock; `harness.gpu_lock` records which. On the cpu lane it records "none (cpu lane, the rule''s clause 2)".' - 'Verbatim: "`evidence/crux/hf-sources.yaml` maps each GGUF (by sha256) to the HF source weights at a PINNED revision." MEASURED: cc324af0… maps to Qwen/Qwen2.5-Coder-1.5B-Instruct@2e1fd397 (bfloat16). Its GGUF provenance was checked through the HF API: the LFS sha256 of qwen2.5-coder-1.5b-instruct-q4_k_m.gguf in Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF@f86cb2c1 equals the local file''s.' - 'Verbatim: "A degenerate answer is no answer, from any engine." MEASURED: case rows cover apr''s `!!!!…` as RED where llama.cpp answered, and hf''s `!!!!…` as unable to vouch. The mutant that removes the rule breaks 2 rows. The table is 35 ok, 0 broke.' - 'Also measured (~21:45Z): a plugin engine is held to its lane as apr is. An hf/llamafile gen row must carry `reported.device` matching its lane (`cpu*` on the cpu lane, anything else on gpu), and a missing device is no answer. This came out of the integration run, where HF on the cpu lane reached the 4090 outside the GPU lock (infra-3c confirmed it and fixed their side in 9c806f008). Rows: hf on the CPU in the gpu lane, and hf with no device, are both UNJUDGED. Each mutant breaks 1. Cells also carry `positive_control` (aprender-97''s additive field for pv). The table is 38 ok, 0 broke.' diff --git a/docs/roadmaps/roadmap.yaml b/docs/roadmaps/roadmap.yaml index 54d57f6e0e..1d0b3940ad 100644 --- a/docs/roadmaps/roadmap.yaml +++ b/docs/roadmaps/roadmap.yaml @@ -19999,7 +19999,7 @@ roadmap: priority: critical assigned_to: aprender-76 created: 2026-09-21T19:20:00Z - updated: 2026-09-21T21:50:00Z + updated: 2026-09-21T22:00:00Z spec: null acceptance_criteria: - 'Issue #3774 done_when 1, verbatim: "`evidence/crux/verb-correspondence.yaml` (`schema: crux-verb-correspondence/v1`) has exactly one entry per LEAF `key` of the surface `apr surface --json` emits (#3745 S1, `apr-cli-surface/v1`), hidden and foreign leaves included, with aliases counted as the same verb. It carries `surface: {source, sha256}` binding it to the emitted surface it was derived from." MEASURED: 253 entries = the 253 leaves of the `apr-cli-surface/v1.1` surface emitted by apr 0.69.0 (7ed033b63, branch PMAT-3745-s1-surface-emitter), sha256 207c3297…da2b. The entry key set equals the leaf key set, with no duplicates. Every `generates: true` leaf that maps no engine (embed, rerank, mcp, parity, rosetta compare-inference) has a specific reason; parity and compare-inference compare apr with apr, and their external comparison is the `run` cells.' @@ -20010,7 +20010,7 @@ roadmap: - 'Verbatim: "`tok` rows ... and `tmpl` rows ... are byte-equal or RED, with the first differing index or byte named. An engine row with no apr row beside it is RED. A row the reference engine refused is UNJUDGED, and the run declines. `greedy` rows are REPORTED." MEASURED rows: equal tok ids are GREEN; differing ids are RED at index 1; an HF tok row without apr is RED; an HF-refused tok row declines rc 2; an equal tmpl is GREEN; the #3672 double wrap is RED at byte 17; greedy divergence at step 2 is reported with its logit cosine 0.7071 and no verdict.' - 'Verbatim: "`scripts/crux_inference_dogfood.sh` exports `CRUX_MANIFEST` and `CRUX_WORK`, resolves `scripts/crux_engine_.{sh,py}` ... A missing script, a failing probe, or a `gen` that appends no row becomes a refused row naming why." MEASURED on lambda with a throwaway fake hf driver (never committed): probe recorded verbatim as `hf.probe`; 3 hf gen rows judged correct; the absent llamafile driver refused by name on every cell; the receipt table has 5 engine columns. The driver still reads no clock (0 rate tokens) and check_no_competing_harnesses stays count=0.' - 'Mutation for the judge side: dropping the tok/tmpl mismatch-or-absent RED breaks 3 rows; not failing the run on a deterministic RED breaks 3; not declining a deterministic UNJUDGED breaks 1; dropping hf as a comparator breaks 1; not requiring the contract `text` field breaks 1; taking the logit cosine at the wrong step breaks 1. The slice-1 mutants still hold.' - - 'Added ~21:15Z, verbatim from #3774: "The GPU lock goes through gpu-q ... Each cell (one prompt, every engine) is written as one script and runs as ONE gpu-q command." MEASURED: the driver has no fd-9 lock left (0 hits for 9>&-/cell_lock/eng_run). On the gpu lane it runs `GPUQ_WAIT=$LOCK_WAIT gpu-q --prio ${CRUX_GPU_PRIO:-5} -- bash ` when `gpu-q --caps` lists `wait` (it does on lambda and gx10), else the bounded flock; `harness.gpu_lock` records which. On the cpu lane it records "none (cpu lane, the rule''s clause 2)".' + - 'Added ~21:15Z, verbatim from #3774: "The GPU lock goes through gpu-q ... Each cell (one prompt, every engine) is written as one script and runs as ONE gpu-q command." MEASURED: the driver has no fd-9 lock left (0 hits for 9>&-/cell_lock/eng_run). On the gpu lane it runs `GPUQ_WAIT=$LOCK_WAIT gpu-q --prio ${CRUX_GPU_PRIO:-3} -- bash ` (the cop''s priority classes: CRUX development = 3, the train''s T-1 run passes 1) when `gpu-q --caps` lists `wait` (it does on lambda and gx10), else the bounded flock; `harness.gpu_lock` records which. On the cpu lane it records "none (cpu lane, the rule''s clause 2)".' - 'Verbatim: "`evidence/crux/hf-sources.yaml` maps each GGUF (by sha256) to the HF source weights at a PINNED revision." MEASURED: cc324af0… maps to Qwen/Qwen2.5-Coder-1.5B-Instruct@2e1fd397 (bfloat16). Its GGUF provenance was checked through the HF API: the LFS sha256 of qwen2.5-coder-1.5b-instruct-q4_k_m.gguf in Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF@f86cb2c1 equals the local file''s.' - 'Verbatim: "A degenerate answer is no answer, from any engine." MEASURED: case rows cover apr''s `!!!!…` as RED where llama.cpp answered, and hf''s `!!!!…` as unable to vouch. The mutant that removes the rule breaks 2 rows. The table is 35 ok, 0 broke.' - 'Also measured (~21:45Z): a plugin engine is held to its lane as apr is. An hf/llamafile gen row must carry `reported.device` matching its lane (`cpu*` on the cpu lane, anything else on gpu), and a missing device is no answer. This came out of the integration run, where HF on the cpu lane reached the 4090 outside the GPU lock (infra-3c confirmed it and fixed their side in 9c806f008). Rows: hf on the CPU in the gpu lane, and hf with no device, are both UNJUDGED. Each mutant breaks 1. Cells also carry `positive_control` (aprender-97''s additive field for pv). The table is 38 ok, 0 broke.' diff --git a/scripts/crux_inference_dogfood.sh b/scripts/crux_inference_dogfood.sh index f053f568d0..6972e2c80f 100755 --- a/scripts/crux_inference_dogfood.sh +++ b/scripts/crux_inference_dogfood.sh @@ -53,7 +53,7 @@ # # GPU sharing (the cop's rule, /mnt/nvme-raid0/agent-wt/gpu-lock-rule.txt, rev 5): # on the gpu lane each cell (one prompt, every engine) runs as ONE command through -# gpu-q (priority queue in front of /tmp/apr-gpu.lock; CRUX_GPU_PRIO, default 5), +# gpu-q (priority queue in front of /tmp/apr-gpu.lock; CRUX_GPU_PRIO, default 3), # falling back to a bounded flock recorded as "may have jumped the queue". Every # engine runs under `choom -n 1000`, so a measurement is the OOM victim, never a # CI runner, and ollama must leave VRAM before the cell ends. @@ -202,7 +202,9 @@ for eng in hf llamafile; do done GPUQ="${GPUQ_BIN:-$HOME/.local/bin/gpu-q}" GPUQ_OK=0 -GPU_PRIO="${CRUX_GPU_PRIO:-5}" +# gpu-q priority classes (the cop, 2026-09-21): CRUX development runs = 3, the +# release train's own T-1 run = 1 (its autopilot passes CRUX_GPU_PRIO=1). +GPU_PRIO="${CRUX_GPU_PRIO:-3}" LOCK_VIA="flock /tmp/apr-gpu.lock (no gpu-q with \`wait\` on this host: may have jumped the queue)" if [ -x "$GPUQ" ]; then gpuq_caps=$("$GPUQ" --caps 2>/dev/null)