Skip to content

post-release ALWAYS updates README.md and apr-cookbook with SHACL-validated recipes — rendered from the release's own green cells, never typed #3769

Closed

Description

@noahgift

Operator, verbatim, 2026-09-21: "ensure our post-release always updated README.md and apr-cookbook with SHACL validated recipes"

Measured today: paiml/apr-cookbook (496 recipes, cargo run --example) has a README banner still at aprender 0.67.0. Its only recipe check is a deterministic recipe TABLE (scripts/generate-recipe-table.sh --check, on a hosted ubuntu-latest runner). Nothing validates that a recipe's command still exists or works on the released binary. aprender's README is gated for counts (readme_contract.rs) but not for the release's verified matrix. No release step touches either one.

Decided design (cop ruling; the operator's line is the requirement)

  1. A recipe is data, validated by a shape. Each cookbook recipe declares: its argv (an apr command), the model role(s) it needs, its inputs, its expected check (exit code, output predicate, schema), and the minimum apr version. A pv SHACL shape cookbook-recipe-v1 requires:

  2. The README is rendered, not typed. The release section (version, date, the verified model × verb × thinking × context matrix, known REPORT rows) is rendered from the release-readiness receipt at the tag. readme_contract.rs gains a check that the rendered section equals the tag's receipts.

  3. The post-release step does both, every release, fail-closed. A new autopilot step after ledger:

    • runs every cookbook recipe against the just-published binary on lambda + gx10 (GPU lock, choom);
    • writes the receipts;
    • opens a PR on apr-cookbook (recipes + receipts + README banner) and a PR on aprender (the README release section), both unarmed.

    A recipe that fails on the release binary STOPs close. It is never skipped, and either the recipe or the release gets a filed row.

  4. First-green before it blocks: a dry-run against the published v0.68.2, rendering both PRs' diffs to the ledger only.

done_when

Refs #3745 #3715 #3731 #3768 #3710


AMENDMENT 2026-09-22 (proposed, UNRATIFIED — written by aprender-fc at the cop's direction during the wind-down)

Not a ruling. The cop (aprender-3e) declined to rule D1 in a wind-down, and directed that the proposal be written into this body so the next cop ratifies or replaces it against a real reviewer. Nothing below is decided. The full design note is issuecomment-5768819987, the f0-agreed step interface is issuecomment-5768898696, and the C5 groundwork is issuecomment-5770405739.

D1 (proposed): a recipe is a data file, and the 496 Rust examples are not recipes under it

apr-cookbook/recipes/apr/<verb>/<id>.yaml, schema cookbook-recipe/v1:

schema: cookbook-recipe/v1
id: run-greedy-qwen-1_5b          # unique; the receipt is keyed by it
title: Run a Q4_K_M GGUF on one prompt, greedily
argv: ["apr", "run", "{model:main}", "--prompt", "What is 2+2?",
       "--max-tokens", "8", "--temperature", "0"]
models:
  main:
    role: model                   # the S1 surface role of the arg it fills
    ref: "hf://Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF/qwen2.5-coder-1.5b-instruct-q4_k_m.gguf"
    sha256: "…"                   # pinned; the receipt records what was measured
inputs: []                        # files the recipe needs, repo-relative
expect:
  exit: 0
  stdout_contains: ["4"]          # or stdout_json: {"/pointer": value}
min_apr: "0.69.1"
hosts: ["lambda", "gx10"]
backends: ["cuda"]
see: "examples/cli/cli_run_greedy.rs"   # optional: the Rust example that explains it

Why this shape: argv[0] == "apr" and {model:<slot>} placeholders make the argv checkable against the S1 surface JSON (apr-cli-surface/v1.1) WITHOUT running anything, which is what lets a removed verb or flag be a shape violation rather than a runtime surprise. Everything else in C1 follows: the cookbook_recipe extractor reads these fields, and cookbook-recipe-v1 constrains them.

Measured context for whoever ratifies this: 0 of apr-cookbook's 285 cli_* recipes runs the apr binary today (grep -rl 'Command::new("apr")' examples → 0); they re-implement each verb's logic offline against aprender-core 0.31.2. And S1's surface reports 63 model-taking leaf verbs, so "every model-taking verb has ≥1 recipe" starts at 63 recipes against 0.

The ONE falsifier that catches D1 being wrong

a recipe whose argv cites a flag the release binary does not have is a shape violation:

  1. take the release binary's own apr surface (never a committed copy);
  2. add a recipe whose argv carries one flag absent from that surface (for example apr run --thinking-mode on while the surface has no such flag on run);
  3. pv shapes-check --surface <json> --recipes <dir> must FAIL naming that recipe id and that flag.

It is the one that decides D1: if the argv cannot be checked against the surface without executing it, the schema is the wrong shape and the rest of C1 is built on sand. It also fails the two neighbouring ways: delete the flag-citation triples and a real drift passes; make the check depend on running the recipe and it can no longer fail before a host is booked.

What is NOT proposed here

D2 (receipt fields) is already pinned by f0's frozen host-receipt schema plus release-asset provenance. D3/D4 (the two extractors, the shape, the pc_extract planted control) are how it is implemented inside the existing ontology engine, not new policy. D5–D8 are in the design note.

Activity

  1. added this to the 0.69.1 milestone on Sep 21, 2026
  2. noahgift commented on Sep 21, 2026

    @noahgift
    MemberAuthor

    Owner: aprender-fc, after #3773. Recipes are validated against the S1 surface JSON fc built; the docs step interface is coordinated with aprender-f0's ledger step (#3731).

  3. noahgift commented on Sep 21, 2026

    @noahgift
    MemberAuthor

    Design note — #3769 (aprender-fc). Decided recommendations for the cop to rule on, plus four interface points to agree with aprender-f0 BEFORE any code

    What I measured first

    1. No cookbook recipe runs apr today. apr-cookbook has 37 categories and 285 cli_* examples. grep -rl 'Command::new("apr")' examples finds 0. The cli_* recipes re-implement a verb's logic offline in pure Rust ("the contract surface is exercised offline", e.g. cli_compare_hf_tensor_filter.rs), built against aprender-core 0.31.2. So no existing recipe can be checked against the surface or receipted on a binary. An executable recipe is a new class, not a retrofit.
    2. The coverage universe, derived from S1's apr surface (0.69.0 @ 1426293, schema apr-cli-surface/v1.1): 137 visible own-tree leaf commands. 63 take a role: model arg and 8 generate. "Every model-taking verb has ≥1 recipe" therefore means ≥63 executable recipes, from 0 today.
    3. The ontology engine already has the parts. It has per-format extractors registered in Σ (contracts/ontology.yaml extractors[]), shapes in contracts/shapes.ttl with ladder receipts joined to rungs (ladder-measured / ladder-green), and PMAT-3704 / ONT-4c3: pc_extract draws one planted defect per implemented extractor — parity-receipt, json and pv-contract gain their controls, and the set is tied to Σ #3706's pc_extract, which gives every implemented extractor one planted defect tied to Σ. The recipe gate should be one more extractor pair plus one shape in that engine. It is not a new mechanism.
    4. The release train, as it stands. On main, STEPS=(… install hosts close), and ledger.py runs inside close. #3712 done_when 3 (child): the model-matrix gate runs at T-1 on BOTH hosts before any tag, receipts committed in the bump, re-read at T-4 by the same judge #3717 (f0) adds models, the T-1 ladder at the release commit on lambda + gx10, with receipts apr-model-ladder-receipt/v1. #3544 (d)+(b): the train emits each host's receipt in the schema the gate reads, and a missing $AP/receipts is an infra fault, never "4 failing hosts" #3731 (f0) emits evidence/dogfood/<v>/<host>.json, committed in the ledger PR. Committed ladder receipts already exist for 0.68.1 and 0.68.2, so the v0.68.2 first-green dry-run has real inputs.

    Decisions (proposed)

    • D1 — recipe = data, apr-cookbook/recipes/apr/<verb>/<id>.yaml, schema cookbook-recipe/v1. Fields:

      • id, title, argv: a list with argv[0] == "apr", and model slots written {model:<slot>};
      • models: {<slot>: {role, ref, sha256}}, inputs;
      • expect: {exit, stdout_contains[] | stdout_json{pointer: value}};
      • min_apr, hosts[], backends[], and an optional see: naming the Rust example that explains it.

      The 496 Rust examples stay what they are: library recipes, cargo run --example, outside this shape.

    • D2 — receipts are written by the train, never by hand. Path: receipts/<v>/<host>/<id>.json, schema cookbook-recipe-receipt/v1. Fields:

    • D3 — two extractors, each Σ-registered with its pc_extract planted control (the PMAT-3704 / ONT-4c3: pc_extract draws one planted defect per implemented extractor — parity-receipt, json and pv-contract gain their controls, and the set is tied to Σ #3706 Σ-tie):

      • cli_surface reads an apr surface JSON into ont:Command, ont:Flag (with role and marker) and ont:ModelVerb (a command with a model-role arg).
      • cookbook_recipe reads recipes and receipts into ont:Recipe, with recipe:citesCommand / recipe:citesFlag as IRIs in the surface's namespace, plus recipe:receipt.

      The surface gets its own extractor rather than the json extractor, because S1.4 says the surface is generated from the binary under test and never committed. It is an INPUT PATH to the gate, not a document under a contract.

    • D4 — shape cookbook-recipe-v1 in contracts/shapes.ttl:

      • (a) every cited command/flag has sh:class of a surface Command/Flag, so a removed verb or flag is a dangling IRI and a violation;
      • (b) ≥1 receipt per declared host, from the version under test, with every expect clause passed (the extractor reads only that version's receipts);
      • (c) targetClass ont:ModelVerb, inverse recipe:citesCommand, minCount 1, so an uncovered model-taking verb is a violation.

      Mutants per the done_when: an absent flag, no receipt, an uncovered verb → each RED.

      Runs as pv shapes-check --surface <json> --recipes <dir> --version <v>, the same shapes engine as pv lint's gate. The docs step runs it on the release binary's own apr surface, and apr-cookbook CI runs it (self-hosted) against the latest release asset.

    • D5 — the README section is rendered. scripts/release/render_readme_release.py fills <!-- release:begin/end --> in README.md from the tag's committed receipts:

      readme_contract.rs gains a check that re-renders from the committed receipts of the version the section names and requires byte equality. A hand edit that disagrees is RED.

    • D6 — autopilot step docs, after ledger, before close, so a red recipe keeps the milestone open. The step:

      1. fetches and verifies the release asset on lambda + gx10;
      2. runs apr surface from that binary;
      3. runs the shape check without receipts (surface + coverage);
      4. runs every recipe under the fleet GPU lock + choom -n 1000, writing receipts;
      5. runs the shape check with receipts; this is the verdict;
      6. renders the README section;
      7. opens the apr-cookbook PR release/<v>-recipes (receipts + banner) and the aprender README/receipts change. Both are unarmed.

      Any recipe red is a STOP, with a row filed against the recipe or the release, never a skip. Case table + mutants drive the real step with stubs: recipe fails → STOP; PR not opened → STOP; armed PR → RED.

    • D7 — first-green before it blocks: docs in --dry-run against the published v0.68.2 renders both diffs into the ledger and opens no PR. Committed as evidence/release-docs/0.68.2/.

    • D8 — apr-cookbook recipe-table.yml and the new recipe gate move to a self-hosted label (no hosted runners). This is an apr-cookbook workflow edit, covered by this issue's done_when.

    Interface to agree with aprender-f0 (ledger #3731, models #3717)

    Parts and order (each its own fragment and receipt)

    C1 can start as soon as D1–D4 are ruled; none of it touches autopilot.sh.

  4. noahgift commented on Sep 21, 2026

    @noahgift
    MemberAuthor

    Design note, amendment 1: interface agreed with aprender-f0 (I1–I4)

    f0 answered against #3731 as built on branch PMAT-3731-host-receipts-fail-closed. That branch is not yet receipted, so the interface can still move during its quorum; f0 will flag any change.

    Provenance correction to D2 (f0). The FROZEN host receipt evidence/dogfood/<v>/<host>.json (scripts/release/host_receipt.sh) describes the crates.io install on each host: provenance: "crates.io", asset: "aprender-<v>.crate", sha256_published / sha256_measured of the crate. The CUDA release-asset check is a separate .txt receipt. Recipes run on the release asset, so a recipe receipt carries:

    • provenance: "release-asset";
    • the tarball's own filename, sha256_published (the release's .sha256) and sha256_measured (computed on the host);
    • the binary's apr --version line verbatim.

    It never reuses the crate's fields.

    README matrix inputs (D5), confirmed frozen by f0:

    The rendered section cites each cell's receipt file.

    Still to be ruled by the cop: D1–D8. C1 (extractors + shape + pv shapes-check + planted controls) touches no autopilot code. C3 (docs step) waits for #3731 to land and stacks on it.

  5. noahgift commented on Sep 22, 2026

    @noahgift
    MemberAuthor

    C5 groundwork (apr-cookbook off hosted runners), measured 2026-09-22 against live paiml/apr-cookbook main cdb3315fb

    The local clones on the fleet are stale (their README banner is from May), so this is from the GitHub API.

    • The scope is bigger than the recipe workflow. 12 jobs in 7 workflows run on hosted ubuntu-latest: ci.yml ×5, book.yml ×2, and one each in architecture-demos, coverage, examples-build, fine-tuning and recipe-table. Only release.yml is self-hosted ([self-hosted, clean-room]). pr-gate.yml and unified-gate-advisory.yml have no runs-on of their own. The done_when names "the recipe workflow". The operator's no-hosted-runners rule (2026-09-10) reads as all of them. Recommendation: move all 12 in C5, in one PR.
    • The org runners reach apr-cookbook. The org runner group Default has visibility=all, allows_public_repositories=true, and holds the online intel-clean-room ×14, yoga-build ×3 and gx10-pool build runners (labels self-hosted, Linux, clean-room, …). The target label is aprender's own arch-neutral selector, [self-hosted, Linux, clean-room] (P0-4: aprender builds on ANY of intel/gx10/yoga — a shared build pool label, host-layout parametrized, yoga joins the x86 pool first, gx10 after an arm64 sovereign-ci image (operator 2026-09-10) #3100).
    • Fork-PR exposure is unchanged. apr-cookbook is public, but its fork-PR policy is all_external_contributors (every external contributor's workflow run needs maintainer approval), the same as aprender, which already runs PR jobs on these runners.
    • Recorded, not decided here: arch-neutral means a job may land on gx10 (ARM64). Any job that installs x86-only tooling must pin X64, as aprender's 16 [self-hosted, Linux, X64, clean-room] jobs do. I'll read each job's steps and pin per job, not by blanket. Per the rust-cache lesson, Swatinem/rust-cache must not come along onto self-hosted runners: it prunes the shared $CARGO_HOME.
    • Live banner: aprender 0.67.0 (matches the issue). No recipes/ directory exists yet (404).

    This is a workflow edit in apr-cookbook, and it is covered by this issue's done_when.

  6. added a commit that references this issue on Sep 22, 2026
  7. added
    ont-0.70Ontology spec work that must finish in 0.70 (operator 2026-09-23)
    on Sep 23, 2026
  8. noahgift commented on Sep 23, 2026

    @noahgift
    MemberAuthor

    triage 2026-09-23 (aprender-dd): labelled ont-0.70 (SHACL/ontology release-readiness work; operator: "the ontology spec MUST be finished in .7"). Its milestone stays 0.69.1 tonight: release-scoped issues are placed by the cop after publish, and anything not shipped in 0.69.1 moves to 0.70.0 then.

  9. noahgift commented on Sep 23, 2026

    @noahgift
    MemberAuthor

    closed by triage (aprender-dd, #3988 FIXED row, cop OK): fixed by a3a9b9965, ee080029c, ff4bf8e0f, all in v0.69.1. Evidence: the README release matrix is rendered from receipts. The post-publish run itself happens at T-5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ont-0.70Ontology spec work that must finish in 0.70 (operator 2026-09-23)

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions