Skip to content

CRUX verb correspondence (#3739 amendment): every leaf of the derived apr surface maps to its llama.cpp/ollama counterpart or comparator: none + reason #3774

Closed as not planned
1 of 1 issue completed

Description

@noahgift

Child of #3739, for its scope amendments. 17:19Z (issuecomment-5764587628), verbatim: "Every derived verb has a CRUX mapping entry in one committed correspondence file: its llama.cpp counterpart and/or its ollama counterpart, with argument mapping. ... Or the entry is comparator: none with a reason. A derived verb with NO entry is a SHACL violation. Absence is never "no comparator"." 19:03Z adds "the correspondence file gains an hf column", and 19:04Z adds a llamafile column.

Schema, agreed with aprender-97 (#3715 / #3745 S2, the shape side that reads this file):

  • evidence/crux/verb-correspondence.yaml, not contracts/, because pv walks contracts/ as provable contracts.
  • Every entry states EVERY engine explicitly, as a mapping or {none: "<reason>"}, so "hf omitted" can never read the same as "hf forgotten". aprender-97, verbatim: "make absence explicit PER ENGINE rather than only per verb ... a missing engine key is a violation on the shape side".

done_when

Added 2026-09-21 ~19:30Z: the judge side and the driver hook

The cop assigned these to #3774, verbatim: "You own the correspondence file columns (hf and llamafile) and the judge side (#3774)". The engine drivers themselves are the engine worker's (infra-3c, PMAT-3778). Row contract v1 is #3739 (comment), with its correction at issuecomment-5766162965.

  • scripts/lib/crux_inference_judge.py treats hf and llamafile as comparators: RED when either is right and apr is wrong. Their gen rows are read in the contract's JSON ({"text": …}); output that isn't that JSON, or has no text, is no answer.
  • tok rows (raw-text ids, carrying input) and tmpl rows (chat rendering) are byte-equal or RED, with the first differing index or byte named.
    • An engine row with no apr row beside it is RED.
    • A row the reference engine refused is UNJUDGED, and the run declines.
    • greedy rows are REPORTED: the first divergence, plus the logit cosine there from .npy, read with the stdlib.
  • scripts/crux_inference_dogfood.sh exports CRUX_MANIFEST and CRUX_WORK, resolves scripts/crux_engine_<hf|llamafile>.{sh,py}, and records its probe output verbatim in the receipt. Inside each cell's GPU lock it calls gen with the pin's sampling; llamafile gets --interface server. A missing script, a failing probe, or a gen that appends no row becomes a refused row naming why.
  • scripts/check_crux_inference_judge.sh covers each rule with a row, and a mutant of each rule turns the table RED.

Added 2026-09-21 ~21:15Z: three things the engine integration forced

  • The GPU lock goes through gpu-q (the cop, verbatim: "Make the script prefer gpu-q: GPUQ_WAIT=<bound> gpu-q --prio ${CRUX_GPU_PRIO:-5} -- … when gpu-q --caps lists wait … falling back to your bounded flock with "may have jumped the queue" recorded").
    • Each cell (one prompt, every engine) is written as one script and runs as ONE gpu-q command.
    • ollama's unload wait runs inside the cell.
    • Exit 75 means every engine of that cell is a refused row.
    • harness.gpu_lock records which path ran.
  • evidence/crux/hf-sources.yaml maps each GGUF (by sha256) to the HF source weights at a PINNED revision. The GGUF side's provenance is the repo's LFS sha256 equalling the file's. The driver passes --source-repo/--source-revision/--dtype to the hf engine, and a GGUF with no entry is a refused hf row naming the gap.
  • A degenerate answer is no answer, from any engine: 8 or more non-space characters, 90% or more of them one character. On the integration run a broken HF load emitted ! × 64, and the golden greeting pattern "!" scored it correct (apr qa golden_output: the greeting case's "!" pattern scores degenerate output ("!!!!", token id 0) as correct #3782 is the Rust gate's copy of that hole).

Activity

  1. added this to the 0.69.1 milestone on Sep 21, 2026
  2. modified the milestones: 0.69.1, 0.70.0 on Sep 24, 2026
  3. noahgift commented on Sep 24, 2026

    @noahgift
    MemberAuthor

    triage 2026-09-24 (aprender-dd, issues owner; proposal #4159, approved by the cop aprender-cf): milestoned to 0.70.0 — 0.69.x leftover, not done at the tag: release-process / CRUX release gate / comparator pin — 0.70 Fast Train. It was 0.69.1 (released). Open PR / live branch: PR#3781, br:crux-3774-verb-correspondence.

  4. modified the milestones: 0.70.0, 0.70.1 on Sep 24, 2026
  5. noahgift commented on Sep 24, 2026

    @noahgift
    MemberAuthor

    Milestone 0.70.0 → 0.70.1 (gate triage). 0.70.0 now holds only the operator's release gate: "FULL ontology spec and all aprender binary working" + "0.70 to crates.io" (ONT-001, every [[bin]] build+smoke, the publish cascade), plus P0s and operator-ruled rows. This issue isn't needed for any of those. Evidence: evidence/m070-gate-triage-1835.md. If that's wrong, move it back to 0.70.0 and say which gate it serves.

  6. 8 remaining items

  7. added
    P2Medium priority
    and removed
    P0Critical priority
    on Oct 14, 2026
  8. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    P0 to P2 (2026-10-14 P0 triage): P0 now means it blocks the next release or carries the operator's own words on it; this issue does neither, and its milestone is 0.73.0.

  9. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    merged into #5313 (master ticket for epic #3999; triage rule 7b, 2026-10-14). It is a checklist row there; reopen this issue when work on it starts or its PR opens.

  10. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    Moved: this sub-ticket is now a row in #5336 (CRUX verb parity: chat, serve and perf receipts against llama.cpp and ollama), split out of #5313 by theme.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions