From 58f784663b5ca47db2a7d8bb326f8265f34dc10f Mon Sep 17 00:00:00 2001 From: Vladimir Rotariu Date: Tue, 27 Oct 2026 01:48:12 -0600 Subject: [PATCH 1/3] docs: render published figures from their sources Every number the README and docs state about the project is now a marker, VALUE, that scripts/render_figures.py renders from the artifact or source that produces it. Figures a routine PR moves are rounded floors ("2,000+"); recorded story pass rates stay exact. check_evidence_claims.py now refuses a figure its checks match outside a marker, and keeps its semantic checks (full-family runs, validated tiers, coverage anchors) running on the rendered text. It also owns the marker grammar and the new source readers for recorded runs. check_public_claims.sh runs render_figures.py --check. The README shows the exact Rust and frontend test counts as live badges read from tests/evidence/workspace-tests.json. The README line "has since grown to 96" was stale (the Ubuntu family holds 99); it is now derived. Two figures sat inside fenced code blocks, where a marker is refused, and were reworded without the number. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU --- CHANGELOG.md | 15 + CONTRIBUTING.md | 19 +- HACKING.md | 8 +- README.md | 20 +- docs/architecture.md | 2 +- docs/contributing/testing.md | 39 ++- docs/contributing/ubuntu-vm-testing.md | 6 +- docs/developer-guide.md | 9 +- docs/distro-support.md | 8 +- docs/introduction.md | 8 +- docs/quickstart.md | 2 +- docs/release-readiness.md | 2 +- docs/testing/user-stories.md | 6 +- docs/typed-actions.md | 2 +- scripts/check_evidence_claims.py | 369 +++++++++++++++++++++++-- scripts/check_public_claims.sh | 8 + scripts/render_figures.py | 188 +++++++++++++ tests/release/public-claims.test.sh | 151 ++++++---- 18 files changed, 750 insertions(+), 112 deletions(-) create mode 100644 scripts/render_figures.py diff --git a/CHANGELOG.md b/CHANGELOG.md index b4b9bc9c..65f29f82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,21 @@ Releases before `0.2.5` predate the public launch; their notes live in the ## [Unreleased] +### Changed + +- **Published figures are rendered from their sources, never typed.** + `scripts/render_figures.py` rewrites every `` marker in the + README and docs (test counts, action counts, story-set sizes, recorded pass + rates), and `scripts/check_evidence_claims.py` now refuses a figure outside a + marker. Figures that routine PRs move are rounded floors ("2,000+"); recorded + story pass rates stay exact. `UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh` + records the evidence and re-renders the docs in one step, and + `scripts/check_public_claims.sh` runs `render_figures.py --check`. The README + shows the exact Rust and frontend test counts as live badges read from + `tests/evidence/workspace-tests.json`. The README line that said the Ubuntu + suite "has since grown to 96" was already stale (it holds 99) and is now a + derived figure. + ## [0.26.1] — 2026-10-24 ### Security diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bf6cada9..06e1b0a0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -65,10 +65,10 @@ table below. | Area | Why it matters | Difficulty | |---|---|---| -| **Ubuntu LTS support** | The current suite is 99 Ubuntu stories. Committed live-VM evidence covers 79 of those Ubuntu stories on each LTS release, with a committed replay twin that reproduces each run: 22.04, 24.04 and 26.04 all at 79/79. The twenty additional stories are not yet included in a committed live-VM run. `ubuntu-vm.sh` accepts `UBUNTU_RELEASE=jammy\|noble\|resolute`. Remaining: story coverage for the cross-family actions, and every Debian-only action has a story. | medium | +| **Ubuntu LTS support** | The current suite is 90+ Ubuntu stories. Committed live-VM evidence covers 79 of those Ubuntu stories on each LTS release, with a committed replay twin that reproduces each run: 22.04, 24.04 and 26.04 all at 79/79. The 20+ additional stories are not yet included in a committed live-VM run. `ubuntu-vm.sh` accepts `UBUNTU_RELEASE=jammy\|noble\|resolute`. Remaining: story coverage for the cross-family actions, and every Debian-only action has a story. | medium | | **Distro detection coverage** | Robust `/etc/os-release` parsing for every release we claim to support. Pure-function tests against real fixture files, no integration mocks. The existing fixtures at the bottom of `crates/sysknife-core/src/distro.rs` show the shape. | easy | | **Action catalogue gaps** | Add a typed action (for example `EnableFirewallZone`). Small and isolated, and every PR carries the policy entry, the risk level and the tests. | easy | -| **E2E story coverage** | Real prompts, real LLM, real daemon. The suite is 153 stories: 54 atomic + 99 Ubuntu. What is left is the cross-family middle: of the action names available on both families, 46 are still untouched by any story, plus 10 Fedora-only and 0 Ubuntu-only ones. See #233 for the clustered map. | medium | +| **E2E story coverage** | Real prompts, real LLM, real daemon. The suite is 150+ stories: 50+ atomic + 90+ Ubuntu. What is left is the cross-family middle: of the action names available on both families, 46 are still untouched by any story, plus 10 Fedora-only and 0 Ubuntu-only ones. See #233 for the clustered map. | medium | | **Fedora Atomic validation** | The action families exist and `DistroId::is_supported()` returns true for Atomic 41 and up. Nobody has run `tests/e2e/atomic-vm.sh` against a current release. Needs Fedora Atomic hardware or a VM host. | tedious | | **Demo recording on real hardware** | Replace the bundled demo GIF with a 30-second recording on real Ubuntu 26.04 with rollback visible. | easy | @@ -168,16 +168,21 @@ manifests. A per-crate run like `cargo test -p sysknife-types` never touches those, so a green crate is not a green board. -**Adding or removing a Rust test moves a published figure.** The suite size lives -in `tests/evidence/workspace-tests.json`, and `README.md`, -`docs/introduction.md` and `docs/distro-support.md` state it in prose. Regenerate -the artifact and update those three files in the same commit: +**Never type a figure.** Test counts, the action count, story-set sizes and +story pass rates are rendered by `scripts/render_figures.py` from the artifact or +source that produces them, and a number the claim checks match outside a marker +fails the build. Add your tests, run the command below, and commit the evidence +file together with whatever it re-rendered: ```sh UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh -grep -rn 'Rust tests' README.md docs/introduction.md docs/distro-support.md ``` +Figures that routine pull requests move are rounded floors ("2,000+"), so most +PRs re-render nothing. The exact Rust and frontend counts are the README badges, +read from the evidence file. The marker syntax is explained once, in +[docs/contributing/testing.md](docs/contributing/testing.md#published-figures). + `workspace-tests.json` stores the expected count for each suite and the canonical command that measures it. Verification re-runs the suite on the current checkout and requires the observed count to match. It makes no historical claim diff --git a/HACKING.md b/HACKING.md index d4502133..77e15b84 100644 --- a/HACKING.md +++ b/HACKING.md @@ -631,12 +631,18 @@ chained to the previous one. short reference version of this file. - [`docs/testing/user-stories.md`](docs/testing/user-stories.md) — the 20 worked Atomic scenarios and their pass criteria (the atomic family itself - is 54 stories; the Ubuntu family is a further 99). + is 50+ stories; the Ubuntu family is a further 90+). - [`tests/e2e/atomic-vm.sh help`](tests/e2e/atomic-vm.sh) — the subcommand reference, kept in sync with the script itself. ## 17. Adding a new action +Published figures (the action count, test counts, story-set sizes) are never typed: +they are `fig` markers rendered by `scripts/render_figures.py`. After adding an +action or a story, run `UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh` and commit +the evidence file and whatever it re-rendered. Syntax: +[docs/contributing/testing.md](docs/contributing/testing.md#published-figures). + An "action" is a named operation the daemon can perform — `GetDiskUsage`, `AddLayeredPackage`, `CreateUser`, etc. Action names are string literals duplicated across several independent match expressions and lists. The diff --git a/README.md b/README.md index c7688fe7..dfb41554 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,8 @@ Discussions npm version Glama MCP server quality score + Rust test count, read from the recorded baseline + Frontend test count, read from the recorded baseline

@@ -234,9 +236,9 @@ privilege boundary is worse than none: sudo make uninstall ``` -All three Ubuntu LTS releases record a live-VM run of 79 stories from the Ubuntu -suite (which has since grown to 96), and each run has a replay twin that reproduces it: 22.04, 24.04 and 26.04 -all at 79/79, every twin serving every call with zero misses. The runs are in +All three Ubuntu LTS releases record a live-VM run of 79 stories from the Ubuntu +suite (which has since grown to 90+), and each run has a replay twin that reproduces it: 22.04, 24.04 and 26.04 +all at 79/79, every twin serving every call with zero misses. The runs are in `tests/evidence/story-runs/`. The suite grew from 50 when every Debian-only action got a story, `GetHostState` first. Fedora Atomic is the rpm-ostree target; `make install` does not complete on @@ -342,18 +344,18 @@ milestone. | Component | State | |---|---| | `sysknife-brain` — LLM planner, tool loop, safety fence | ✅ | -| `sysknife-daemon` — 192 typed actions, auth, preview, transactions | ✅ | +| `sysknife-daemon` — 190+ typed actions, auth, preview, transactions | ✅ | | Live IPC + streaming + atomic-host rollback (rpm-ostree) | ✅ | | Terminal approval gate — one-time, TTL-bounded receipts | ✅ | | MCP server (Claude Code / Cursor / any MCP client) | ✅ | | Tamper-evident Ed25519-signed audit chain | ✅ | | RFC 5424 syslog forwarding (Splunk / Sentinel / QRadar) | ✅ | | Postgres backend (RDS / Cloud SQL / Neon / Supabase) | ✅ | -| **Ubuntu support** — 79/79 stories on a live 22.04 VM, recorded in `tests/evidence/story-runs/` | ✅ | -| **Every Ubuntu LTS validated** — 22.04, 24.04 and 26.04 all at 79/79, each with a replay twin that reproduces it | ✅ | +| **Ubuntu support** — 79/79 stories on a live 22.04 VM, recorded in `tests/evidence/story-runs/` | ✅ | +| **Every Ubuntu LTS validated** — 22.04, 24.04 and 26.04 all at 79/79, each with a replay twin that reproduces it | ✅ | | Telegram approval interface | 📋 roadmap | -**2,004 Rust tests and 72 frontend tests** form the current deterministic +**2,000+ Rust tests and 70+ frontend tests** form the current deterministic release baseline. ## Configure your LLM @@ -399,8 +401,8 @@ All config files that may contain API keys are created with `chmod 0600`. See [ROADMAP.md](ROADMAP.md) for the full milestone breakdown. -- ✅ **Ubuntu 22.04** — 79/79 stories on a live VM (recorded in `tests/evidence/story-runs/`) -- ✅ **Ubuntu 24.04 and 26.04** — 79/79 and 79/79 on live VMs; every LTS run has a replay twin that reproduces it +- ✅ **Ubuntu 22.04** — 79/79 stories on a live VM (recorded in `tests/evidence/story-runs/`) +- ✅ **Ubuntu 24.04 and 26.04** — 79/79 and 79/79 on live VMs; every LTS run has a replay twin that reproduces it - ✅ `sysknife audit export` — stored signed chain rows as JSON with `--since` / `--limit` - 📋 Telegram inline-button approvals - 📋 CEF / NDJSON output modes for SIEM ingest diff --git a/docs/architecture.md b/docs/architecture.md index e89c55ed..c1ead602 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -72,7 +72,7 @@ component uses the same resolution order. Privileged. The only component that touches the system. Provides: -- 192 typed actions (rpm-ostree, systemd, firewall, users, containers, +- 190+ typed actions (rpm-ostree, systemd, firewall, users, containers, flatpak, toolbox, SSH, kernel args, …) - Role-based authorization (`Observer` → `Dev` → `Admin`, plus `Boot`) - Policy enforcement: stale-approval detection, request hash validation diff --git a/docs/contributing/testing.md b/docs/contributing/testing.md index c4ae80ec..6df14fe7 100644 --- a/docs/contributing/testing.md +++ b/docs/contributing/testing.md @@ -13,7 +13,7 @@ validation before a release. | Integration (Rust) | Daemon IPC, safety fence, policy | <10s | Every commit, every CI run | | **Dev stories (local, no VM)** | LLM plan structure for read-only stories; runs on any Linux host | 1-3 min | After brain/prompt changes | | CI smoke (container) | Daemon + Ollama + read-only stories in a Linux runner | 5-10 min | Opt-in (manual `workflow_dispatch` only) | -| E2E Atomic VM | **Real Silverblue** in QEMU/KVM, full stack, all 54 stories | 15-30 min first boot; 2-3 min subsequent | Local / pre-release | +| E2E Atomic VM | **Real Silverblue** in QEMU/KVM, full stack, all 50+ stories | 15-30 min first boot; 2-3 min subsequent | Local / pre-release | | **E2E Ubuntu VM** | **Ubuntu 24.04** cloud image, full story suite (pass rate in `tests/evidence/story-runs/`) | ~15 min first boot; ~2 min subsequent | After Ubuntu action changes | | Manual QA | Real Silverblue/Kinoite hardware, destructive actions, GUI | 30-60 min | Before releases + demo video | @@ -275,7 +275,7 @@ sudo chmod +r /boot/vmlinuz-* # Run the read-only stories (non-destructive default) ./tests/e2e/atomic-vm.sh run -# Run ALL 54 stories including destructive — restore the baseline afterwards. +# Run ALL stories including destructive — restore the baseline afterwards. SYSKNIFE_ALLOW_DESTRUCTIVE=1 ./tests/e2e/atomic-vm.sh run # Roll back to the clean baseline so the next run is fast @@ -376,12 +376,43 @@ sudo -E tests/e2e/run-stories.sh 1 4 7 Per-story logs are written to `tests/e2e/logs/story-N.log`. +## Published figures + +Never type a figure. A count the README or the docs state about the project +(tests, actions, story-set sizes, recorded pass rates) is rendered from its +source by `scripts/render_figures.py`, and `scripts/check_evidence_claims.py` +refuses one that sits outside a marker. After you add tests, run +`UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh` (add `--frontend` for the +vitest suite), then commit the evidence file and whatever it re-rendered. + +A marker is an HTML comment pair around the value, invisible on GitHub and in +mdBook: + +- `VALUE` renders the exact value with thousands + separators. Story pass rates read from a recorded run are exact. +- `VALUE` renders the value rounded down to + a multiple of `K` with a trailing plus. A figure a routine PR moves is a + floor, so most PRs change no prose. `render_figures.py` refuses a floor above + the value. + +`NAME` is a key in the registry at the top of `scripts/render_figures.py`; each +one reads its source through a function from `check_evidence_claims.py`. A +marker sits on one line, cannot nest, and cannot sit inside a fenced code +block. Inline code spans are ignored, which is how this section shows the +syntax. `scripts/render_figures.py --check` names the file and line of every +stale, unknown or malformed marker, and `--write` fixes the stale ones. + +The exact Rust and frontend test counts are the two README badges, which read +`tests/evidence/workspace-tests.json` from `main`. + ## Before opening a PR 1. `cargo nextest run --workspace && npm test --prefix apps/sysknife-shell` — required, fast 2. `cargo clippy --workspace --all-features --all-targets --locked -- -D warnings` 3. `cargo fmt --all --check` -4. For changes to the brain, daemon, IPC, or action catalogue: +4. If you added or removed tests, `UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh` + and commit what it changed (see [Published figures](#published-figures)) +5. For changes to the brain, daemon, IPC, or action catalogue: - Run the VM tests locally (`atomic-vm.sh` flow) - Optionally run the CI smoke test by dispatching the `e2e` workflow @@ -393,7 +424,7 @@ The maintainer runs these in order: 2. VM tests (at least Silverblue + one other atomic variant) pass locally 3. Manual QA on real Silverblue hardware using [docs/testing/user-stories.md](../testing/user-stories.md) as the - checklist — all 54 stories including destructive ones + checklist — all 50+ stories including destructive ones 4. Record the demo video on real hardware (issue #32) ## Troubleshooting diff --git a/docs/contributing/ubuntu-vm-testing.md b/docs/contributing/ubuntu-vm-testing.md index 328da748..2f1b370d 100644 --- a/docs/contributing/ubuntu-vm-testing.md +++ b/docs/contributing/ubuntu-vm-testing.md @@ -7,9 +7,9 @@ Supported Ubuntu LTSes: | Release | Codename | SSH port | Live suite | Status | |---|---|---|---|---| -| 22.04 | jammy | 2222 | 79/79 | validated | -| 24.04 | noble | 2223 | 79/79 | validated (default) | -| 26.04 | resolute | 2224 | 79/79 | validated | +| 22.04 | jammy | 2222 | 79/79 | validated | +| 24.04 | noble | 2223 | 79/79 | validated (default) | +| 26.04 | resolute | 2224 | 79/79 | validated | Each release has a record run and a `.replay.json` twin in `tests/evidence/story-runs/`, and `tests/release/cassette-replay-parity.test.sh` diff --git a/docs/developer-guide.md b/docs/developer-guide.md index 32905396..c8036ba9 100644 --- a/docs/developer-guide.md +++ b/docs/developer-guide.md @@ -282,6 +282,13 @@ and the workspace test baseline. `scripts/test_baseline.sh` runs `cargo nextest run --workspace --locked` and checks the measured test count. Run `bash .githooks/pre-commit` to invoke the same gate manually. +Never type a figure into the README or the docs. Add your tests, run +`UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh`, and commit the evidence file +and whatever it re-rendered. The command records the count and then runs +`scripts/render_figures.py --write`, which rewrites every `fig` marker from its +source. The marker syntax is explained in +[docs/contributing/testing.md](contributing/testing.md#published-figures). + The secret screen recognizes the credential formats of every key-bearing provider in the setup wizard, including OpenAI service-account and `None` keys, DeepSeek's hexadecimal keys, and xAI keys. Mistral's keys have no @@ -367,7 +374,7 @@ crates/ sysknife-brain/ LLM planner, provider adapters, safety fence sysknife-types/ Shared domain types (CallerRole, RiskLevel, JobState, …) sysknife-core/ Config loading, shared constants - sysknife-daemon/ Privileged executor, 191 actions with an `ActionSpec`, + sysknife-daemon/ Privileged executor, one `ActionSpec` per action, IPC, rollback, SQLite sysknife-proto/ Protobuf definitions (future use) sysknife-daemon-test/ TAP integration harness for the daemon IPC layer (VM only) diff --git a/docs/distro-support.md b/docs/distro-support.md index 9b8e184c..c53e67ec 100644 --- a/docs/distro-support.md +++ b/docs/distro-support.md @@ -56,9 +56,9 @@ tracked separately in [#239](https://github.com/lacs-project/sysknife/issues/239 | Distro | Action backend | Evidence | Launch tier | |---|---|---|---| -| **Ubuntu 24.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-24.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok` | **Validated** | -| **Ubuntu 22.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-22.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok`. This is the release whose twin exercises the recorded-rejection path: story 101's first call was refused by the provider (`tool_use_failed`), and the replay serves the refusal, the correction and the answer, 81 calls for 79 stories | **Validated** | -| **Ubuntu 26.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-26.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok`; plus sudo-rs sudoers verification (26.04 ships sudo-rs 0.2.x; `visudo -cf` parses the SysKnife sudoers and every grant — including the trailing-`*` wildcard grants — is honoured) | **Validated** | +| **Ubuntu 24.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-24.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok` | **Validated** | +| **Ubuntu 22.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-22.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok`. This is the release whose twin exercises the recorded-rejection path: story 101's first call was refused by the provider (`tool_use_failed`), and the replay serves the refusal, the correction and the answer, 81 calls for 79 stories | **Validated** | +| **Ubuntu 26.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-26.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok`; plus sudo-rs sudoers verification (26.04 ships sudo-rs 0.2.x; `visudo -cf` parses the SysKnife sudoers and every grant — including the trailing-`*` wildcard grants — is honoured) | **Validated** | | **Every other Ubuntu 20.04+ release** (20.04, 20.10, 21.x, 23.x, 25.x, 26.10, …) | Ubuntu/apt family | Eligible by release family; no per-release VM run | **Smoke-tested** | | **Fedora Silverblue 44** | rpm-ostree, Flatpak, toolbox, firewalld, systemd, containers | Harness and fixture coverage; no live-VM run on this release | **Blocked** (`make install` does not complete on rpm-ostree, [#301](https://github.com/lacs-project/sysknife/issues/301)) | | **Other Fedora Atomic 41+ variants** | rpm-ostree family | Detection and shared action tests, plus a live VM run on Fedora 43 Silverblue (2026-08-24) that provisioned and then stopped at `make install` | **Blocked** (`make install` does not complete on rpm-ostree, [#301](https://github.com/lacs-project/sysknife/issues/301)) | @@ -94,7 +94,7 @@ family and the atomic story family are implemented and covered by the workspace suite. What is missing is a way to put the helpers somewhere the daemon's own grants already point. -The deterministic workspace baseline is 2,004 Rust tests plus 72 frontend +The deterministic workspace baseline is 2,000+ Rust tests plus 70+ frontend tests. Those tests verify action construction, policy, approval, storage, and UI behavior, but they do not replace a real distribution VM run. diff --git a/docs/introduction.md b/docs/introduction.md index 8e61827a..d181cf3b 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -106,10 +106,10 @@ SysKnife, not an afterthought. > **ℹ️ Distro support** > -> The current suite is 99 Ubuntu stories. All three Ubuntu LTS releases have a -> committed live-VM run covering 79 of those Ubuntu stories, in +> The current suite is 90+ Ubuntu stories. All three Ubuntu LTS releases have a +> committed live-VM run covering 79 of those Ubuntu stories, in > `tests/evidence/story-runs/`: -> 22.04, 24.04 and 26.04 all at 79/79. The twenty additional stories are not yet +> 22.04, 24.04 and 26.04 all at 79/79. The 20+ additional stories are not yet > included in a committed live-VM run. Each run has a replay twin that > reproduces it, serving every call with zero misses. > Every Debian-only action has a story. @@ -146,7 +146,7 @@ flow. ## Status -192 typed actions · 2,004 Rust tests + 72 frontend tests · MIT +190+ typed actions · 2,000+ Rust tests + 70+ frontend tests · MIT SysKnife is the reference implementation of the [LACS specification](https://github.com/lacs-project/specification) — a diff --git a/docs/quickstart.md b/docs/quickstart.md index 5957d706..6ba53872 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -72,7 +72,7 @@ build (measured 6m56s on Ubuntu 24.04, 11m43s on 22.04). > **ℹ️ Fedora / Silverblue** > > Ubuntu 22.04, 24.04 and 26.04 are each validated by a committed live-VM -> story run (79/79) with a replay twin; other Ubuntu releases from 20.04 up are +> story run (79/79) with a replay twin; other Ubuntu releases from 20.04 up are > eligible without a per-release run. Fedora Atomic uses the rpm-ostree action > family and requires a current Silverblue 44 validation run for each release. > Plain Fedora remains experimental. See [distro support](distro-support.md). diff --git a/docs/release-readiness.md b/docs/release-readiness.md index 3d614979..de77ee24 100644 --- a/docs/release-readiness.md +++ b/docs/release-readiness.md @@ -12,7 +12,7 @@ Run these on the exact candidate commit and retain the CI links: - [ ] `cargo fmt --all --check` - [ ] `cargo clippy --workspace --all-features --all-targets --locked -- -D warnings` - [ ] `cargo nextest run --workspace --locked` -- [ ] frontend typecheck, 72 tests, production build, and production npm audit +- [ ] frontend typecheck, 70+ tests, production build, and production npm audit - [ ] live PostgreSQL migration and store contract - [ ] ShellCheck, YAML lint, repository completeness, release versions, public claims, and bootstrap contracts diff --git a/docs/testing/user-stories.md b/docs/testing/user-stories.md index e1d1df8f..7ffdc39f 100644 --- a/docs/testing/user-stories.md +++ b/docs/testing/user-stories.md @@ -5,8 +5,8 @@ Fedora Atomic Desktop (Silverblue, Kinoite, Sway Atomic, Budgie Atomic, or COSMIC Atomic). Run inside a QEMU/KVM VM via `tests/e2e/atomic-vm.sh`, or on real hardware. -These twenty are a readable subset, not the suite: the atomic family is 54 -stories and the Ubuntu family a further 99 (79 of them in the recorded +These twenty are a readable subset, not the suite: the atomic family is 50+ +stories and the Ubuntu family a further 90+ (79 of them in the recorded live-VM runs), all in `tests/e2e/stories/`. The harness runs the family; this file explains twenty of them in prose. @@ -500,7 +500,7 @@ by the manual QA checklist (see `demo-script.md`): ## Running the stories -### Locally (real Silverblue VM, all 54 atomic stories) +### Locally (real Silverblue VM, all 50+ atomic stories) ```sh # One-time: download the Fedora Silverblue ISO and install it in QEMU/KVM diff --git a/docs/typed-actions.md b/docs/typed-actions.md index a19cdd04..d4e8a5ac 100644 --- a/docs/typed-actions.md +++ b/docs/typed-actions.md @@ -62,7 +62,7 @@ codebase. On the daemon side (`sysknife-daemon`), each action is backed by an | `reboot_required` | Whether the daemon should warn the caller before proceeding | | `rollback_available` | Whether a failure triggers automatic rollback | -As of this writing the catalogue defines **192 actions** across families such +As of this writing the catalogue defines **190+ actions** across families such as Deployment, Services, Package Layering, Flatpak, Containers, Toolbox, Network, Identity, SSH Keys, Package Repositories, apt/snap/ufw/netplan/grub and AppArmor/cloud-init (Debian-family), Ubuntu Pro (Ubuntu-only), and diff --git a/scripts/check_evidence_claims.py b/scripts/check_evidence_claims.py index b1bc6e46..2c3c145e 100755 --- a/scripts/check_evidence_claims.py +++ b/scripts/check_evidence_claims.py @@ -18,6 +18,20 @@ A story claim additionally has to name a *full* family run. Subset runs record an empty `story_set`, so a four-story probe can never be quoted as the headline. +Figures are not typed. Every figure these checks match in CLAIM_FILES must sit +inside a marker that scripts/render_figures.py owns: + + VALUE exact ("2,004") + VALUE rounded floor ("2,000+") + +This module refuses a matched figure outside a marker (check_unmarked_figures), +so a hand-typed number cannot creep back, and it keeps the semantic checks +(a story claim must name a full family run, validated tiers, ...) running on the +rendered text, with the markers unwrapped. Freshness of the marked values is +render_figures.py --check; `UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh` +re-renders them. The marker grammar lives here, once, and render_figures.py +imports it together with the value readers below. + Run via scripts/check_public_claims.sh; tests/release/public-claims.test.sh mutates each claim and asserts this rejects it. """ @@ -119,6 +133,231 @@ def read_claim_files(root: Path) -> dict[str, str]: ) return texts +# --------------------------------------------------------------------------- +# Figure markers. The grammar has one implementation, here; render_figures.py +# imports it. +# +# VALUE exact +# VALUE rounded floor, rendered "N+" +# +# HTML comments are invisible on GitHub and in mdBook. A marker sits on one line. +# Markers inside a fenced code block are refused (a fence is shown literally, so +# the comments would be visible and the value would never be read as a figure); +# markers inside an inline code span are documentation of the syntax and are +# ignored. +# --------------------------------------------------------------------------- +MARKER_START = re.compile(r"") +MARKER_OPEN_REST = re.compile( + r"^:(?P[A-Za-z0-9_.:-]+)(?:\s+floor=(?P[0-9]+))?$" +) +CODE_FENCE = re.compile(r"^\s*(```|~~~)") +INLINE_CODE = re.compile(r"`[^`\n]*`") + + +class Marker: + """One well-formed marker: where it sits and what it currently says.""" + + __slots__ = ("name", "floor", "start", "end", "value_start", "value_end", "line", "value") + + def __init__(self, name, floor, start, end, value_start, value_end, line, value): + self.name = name + self.floor = floor + self.start = start + self.end = end + self.value_start = value_start + self.value_end = value_end + self.line = line + self.value = value + + +def find_markers(text: str) -> tuple[list[Marker], list[tuple[int, str]]]: + """Return the well-formed markers and every structural problem as (line, message). + + Problems: a marker inside a code fence, an unparseable marker comment, an + unbalanced or nested pair, a pair split across lines, a floor of zero. + """ + problems: list[tuple[int, str]] = [] + tokens = [] # (is_close, name, floor, start, end, line) + in_fence = False + offset = 0 + for lineno, line in enumerate(text.splitlines(keepends=True), 1): + line_start = offset + offset += len(line) + if CODE_FENCE.match(line): + in_fence = not in_fence + continue + if in_fence: + if MARKER_START.search(line): + problems.append((lineno, "figure marker inside a fenced code block")) + continue + spans = [m.span() for m in INLINE_CODE.finditer(line)] + for start in MARKER_START.finditer(line): + if any(a <= start.start() < b for a, b in spans): + continue + token = MARKER_TOKEN.match(line, start.start()) + if token is None: + problems.append((lineno, "malformed figure marker (no closing '-->')")) + continue + rest = token.group("rest") + if token.group("close"): + if rest.strip(): + problems.append((lineno, f"malformed closing marker {token.group(0)!r}")) + continue + tokens.append((True, None, None, line_start + token.start(), + line_start + token.end(), lineno)) + continue + opened = MARKER_OPEN_REST.match(rest.strip()) + if opened is None: + problems.append((lineno, f"malformed figure marker {token.group(0)!r}")) + continue + floor = opened.group("floor") + if floor is not None and int(floor) == 0: + problems.append((lineno, "floor=0 is not a rounding")) + continue + tokens.append((False, opened.group("name"), + int(floor) if floor is not None else None, + line_start + token.start(), line_start + token.end(), lineno)) + + markers: list[Marker] = [] + pending = None + for token in tokens: + is_close, name, floor, start, end, lineno = token + if not is_close: + if pending is not None: + problems.append((lineno, f"nested figure marker inside the one opened on line {pending[5]}")) + pending = token + continue + pending = token + continue + if pending is None: + problems.append((lineno, "closing figure marker without an opening one")) + continue + _, name, floor, open_start, open_end, open_line = pending + pending = None + value = text[open_end:start] + if "\n" in value: + problems.append((open_line, "figure marker spans lines")) + continue + markers.append(Marker(name, floor, open_start, end, open_end, start, open_line, value)) + if pending is not None: + problems.append((pending[5], "unbalanced figure marker: opened and never closed")) + return markers, problems + + +def unwrap_markers(text: str) -> str: + """The text a reader sees: each marker replaced by its value.""" + markers, _ = find_markers(text) + out, last = [], 0 + for marker in markers: + out.append(text[last : marker.start]) + out.append(marker.value) + last = marker.end + out.append(text[last:]) + return "".join(out) + + +def mask_markers(text: str) -> str: + """The text with every marker replaced by a placeholder, so what is left is + exactly the figures nothing owns.""" + markers, _ = find_markers(text) + out, last = [], 0 + for marker in markers: + out.append(text[last : marker.start]) + out.append("FIG") + last = marker.end + out.append(text[last:]) + return "".join(out) + + +# The README badge URLs are not figures and need no exemption: they are +# percent-encoded (`label=Rust%20tests`), so no shape below can match inside one. +# The pristine tree passing with both badges present is the proof. +# Digits followed by a floor's trailing plus ("2,000+ Rust tests"): read as the +# bare number so the shapes below match a hand-typed floor exactly as they match +# a hand-typed exact figure. +FLOOR_PLUS = re.compile(r"(?<=[0-9])\+(?=[\s-])") + +# The shapes the checks below match. check_figure builds its pattern from these +# nouns; the rest are the patterns of the check named beside each one. +FIGURE_NOUNS = ("Rust tests", "frontend tests", "typed actions") +BARE_TOTAL_RE = re.compile(r"([0-9][0-9,]*)\s+tests\b") # check_bare_test_totals +STORY_RATE_RE = re.compile(r"([0-9]+)\s*/\s*([0-9]+)\s+stories") # check_story_claims +BARE_STORY_RE = re.compile(r"(?[0-9]{2,})\s+(?Patomic|ubuntu)\b", re.IGNORECASE +) # check_bare_story_counts +ACTION_BARE_RE = re.compile(r"\b([0-9]{2,})\s+(?:typed\s+)?actions\b", re.IGNORECASE) +ACTION_SUBSET_RE = re.compile( + r"\b([0-9]+)\s+actions\s+(?:with|have)\s+an\s+`?ActionSpec`?\b", re.IGNORECASE +) # check_action_figures +# check_story_coverage_claims reads these three counts out of one sentence; each +# is a figure on its own. +UNCOVERED_FIGURE_RES = ( + re.compile(r"\b[0-9]+\s+are\s+still\s+untouched\s+by\s+any\s+story\b", re.IGNORECASE), + re.compile(r"\b[0-9]+\s+Fedora-only\s+and\b", re.IGNORECASE), + re.compile(r"\b[0-9]+\s+Ubuntu-only\s+ones\b", re.IGNORECASE), +) + + +def _figure_shapes(text: str) -> list[tuple[int, int]]: + """Every (start, end) span any figure check would match in `text`.""" + hits: list[tuple[int, int]] = [] + for noun in FIGURE_NOUNS: + spaced = r"\s+".join(re.escape(word) for word in noun.split()) + for match in re.finditer(rf"([0-9][0-9,]*)\s+{spaced}", text): + hits.append(match.span()) + for match in BARE_TOTAL_RE.finditer(text): + if int(match.group(1).replace(",", "")) >= BARE_TOTAL_FLOOR: + hits.append(match.span()) + for regex in (STORY_RATE_RE, BARE_STORY_RE, FAMILY_RE, ACTION_BARE_RE, ACTION_SUBSET_RE, + *UNCOVERED_FIGURE_RES): + for match in regex.finditer(text): + hits.append(match.span()) + return hits + + +def check_unmarked_figures(raw_texts: dict[str, str]) -> list[str]: + """Refuse a figure that the checks above would match but no marker owns. + + The marked figures belong to render_figures.py --check. Without this rule a + hand-typed number would still be validated by the semantic checks, but it + would be one more number a contributor has to edit by hand, and a floor typed + by hand would not be validated at all (a floor is skipped by those checks, as + `2,000+` is not an exact count). + """ + problems = [] + for rel, text in raw_texts.items(): + _, structural = find_markers(text) + for lineno, message in structural: + problems.append(f"{rel}:{lineno}: {message}") + masked = mask_markers(text) + scanned = FLOOR_PLUS.sub("", masked) + # Offsets in `scanned` map back into `masked` by adding the floor-plus + # characters removed before them, so a hand-typed "2,000+" is quoted as typed. + removed = [m.start() for m in FLOOR_PLUS.finditer(masked)] + + def original(offset: int) -> int: + for index, position in enumerate(removed): + if position - index >= offset: + return offset + index + return offset + len(removed) + + seen = set() + for start, end in sorted(_figure_shapes(scanned)): + lineno = scanned.count("\n", 0, start) + 1 + matched = masked[original(start) : original(end)] + key = (lineno, " ".join(matched.split())) + if key in seen: + continue + seen.add(key) + problems.append( + f"{rel}:{lineno}: figure {' '.join(matched.split())!r} is not inside a " + " marker. Do not type figures: wrap it and run " + "scripts/render_figures.py --write" + ) + return problems + def count_actions(root: Path) -> int: """Count entries in the brain's KNOWN_ACTIONS table.""" @@ -167,9 +406,8 @@ def check_figure(texts: dict[str, str], noun: str, expected: int) -> list[str]: def check_bare_test_totals(texts: dict[str, str], allowed: tuple[int, ...]) -> list[str]: """Flag an unqualified ` tests` that looks like a stale workspace total.""" problems = [] - pattern = re.compile(r"([0-9][0-9,]*)\s+tests\b") for rel, text in texts.items(): - for match in pattern.finditer(text): + for match in BARE_TOTAL_RE.finditer(text): # Skip the qualified forms; check_figure owns those. preceding = text[max(0, match.start() - 12) : match.start()] claimed = int(match.group(1).replace(",", "")) @@ -211,10 +449,9 @@ def story_run_has_no_unproven_results(run: dict) -> bool: def check_story_claims(texts: dict[str, str], runs: list[dict]) -> list[str]: problems = [] - pattern = re.compile(r"([0-9]+)\s*/\s*([0-9]+)\s+stories") for rel, text in texts.items(): for line in text.splitlines(): - match = pattern.search(line) + match = STORY_RATE_RE.search(line) if not match: continue passed, total = int(match.group(1)), int(match.group(2)) @@ -470,8 +707,8 @@ def check_story_coverage_claims(texts: dict[str, str], root: Path) -> list[str]: problems = [] family_claim = re.search( - r"\bThe suite is\s+(?P\d+)\s+stories?\s*:\s*" - r"(?P\d+)\s+atomic\s*\+\s*(?P\d+)\s+Ubuntu\b", + r"\bThe suite is\s+(?P\d+\+?)\s+stories?\s*:\s*" + r"(?P\d+\+?)\s+atomic\s*\+\s*(?P\d+\+?)\s+Ubuntu\b", text, re.IGNORECASE, ) @@ -481,17 +718,19 @@ def check_story_coverage_claims(texts: dict[str, str], root: Path) -> list[str]: "'The suite is N stories: A atomic + U Ubuntu'" ) else: + # A rounded floor ("150+") is owned by render_figures.py --check; only an + # exact count is compared here. published_sizes = { - "total": int(family_claim.group("total")), - "atomic": int(family_claim.group("atomic")), - "ubuntu": int(family_claim.group("ubuntu")), + label: int(family_claim.group(label)) + for label in ("total", "atomic", "ubuntu") + if not family_claim.group(label).endswith("+") } derived_sizes = { "total": total, "atomic": family_sizes.get("atomic", 0), "ubuntu": family_sizes.get("ubuntu", 0), } - for label in ("total", "atomic", "ubuntu"): + for label in published_sizes: if published_sizes[label] != derived_sizes[label]: problems.append( f"{rel}: story coverage {label} count has published " @@ -611,10 +850,8 @@ def check_bare_story_counts( individually legitimate sizes is not enough, since swapped labels would otherwise pass. """ - bare = re.compile(r"(?[0-9]{2,})\s+(?Patomic|ubuntu)\b", re.IGNORECASE - ) + bare = BARE_STORY_RE + family = FAMILY_RE allowed = { run.get("totals", {}).get("total") @@ -684,11 +921,8 @@ def check_action_figures( writable, forces it to carry its qualifier, and still catches the count drifting. """ - bare = re.compile(r"\b([0-9]{2,})\s+(?:typed\s+)?actions\b", re.IGNORECASE) - subset = re.compile( - r"\b([0-9]+)\s+actions\s+(?:with|have)\s+an\s+`?ActionSpec`?\b", - re.IGNORECASE, - ) + bare = ACTION_BARE_RE + subset = ACTION_SUBSET_RE problems = [] for rel, text in texts.items(): @@ -814,6 +1048,90 @@ def load_test_baseline(root: Path) -> dict: return baseline +LTS_RELEASES = ("22.04", "24.04", "26.04") + + +def story_run_totals(root: Path, stem: str) -> dict: + """The totals of one recorded run, addressed by its file name stem.""" + path = root / STORY_RUNS / f"{stem}.json" + if not path.is_file(): + raise Failure(f"no recorded story run {STORY_RUNS}/{stem}.json") + try: + run = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise Failure(f"{STORY_RUNS}/{stem}.json is not readable JSON: {exc}") from exc + totals = run.get("totals") if isinstance(run, dict) else None + if not isinstance(totals, dict) or not all( + isinstance(totals.get(f), int) and not isinstance(totals.get(f), bool) + for f in ("passed", "total") + ): + raise Failure(f"{STORY_RUNS}/{stem}.json has no integer totals.passed and totals.total") + return totals + + +def story_run_rate(root: Path, stem: str) -> str: + """`passed/total` of one recorded run, read from its artifact.""" + totals = story_run_totals(root, stem) + return f"{totals['passed']}/{totals['total']}" + + +def story_run_total(root: Path, stem: str) -> int: + """Stories in one recorded run, read from its artifact.""" + return story_run_totals(root, stem)["total"] + + +def story_run_served(root: Path, stem: str) -> int: + """Calls a replay served, from its `cassette_audit`.""" + path = root / STORY_RUNS / f"{stem}.json" + story_run_totals(root, stem) # missing or unreadable: fail with the reader's message + audit = json.loads(path.read_text(encoding="utf-8")).get("cassette_audit") + served = audit.get("served") if isinstance(audit, dict) else None + if not isinstance(served, int) or isinstance(served, bool): + raise Failure(f"{STORY_RUNS}/{stem}.json has no integer cassette_audit.served") + return served + + +def lts_story_rates(root: Path) -> dict[str, tuple[int, int]]: + """`(passed, total)` of the latest full-family record run of each Ubuntu LTS.""" + latest: dict[str, tuple[str, tuple[int, int]]] = {} + for run in load_story_runs(root): + release = str(run.get("release", "")) + if ( + release in LTS_RELEASES + and run.get("story_set") in FULL_STORY_SETS + and run.get("cassette_mode") == "record" + and story_run_has_no_unproven_results(run) + ): + totals = run.get("totals", {}) + key = str(run.get("ran_at", "")) + if release not in latest or key > latest[release][0]: + latest[release] = (key, (totals.get("passed"), totals.get("total"))) + missing = [release for release in LTS_RELEASES if release not in latest] + if missing: + raise Failure( + "no full-family record run is committed for Ubuntu " + ", ".join(missing) + ) + return {release: latest[release][1] for release in LTS_RELEASES} + + +def lts_story_total(root: Path) -> int: + """The story count every LTS run covers. Refuses when the runs disagree.""" + rates = lts_story_rates(root) + totals = {total for _, total in rates.values()} + if len(totals) != 1: + raise Failure(f"the LTS runs cover different story counts: {rates}") + return totals.pop() + + +def lts_story_rate(root: Path) -> str: + """`passed/total` shared by every LTS run. Refuses when the runs disagree.""" + rates = lts_story_rates(root) + if len(set(rates.values())) != 1: + raise Failure(f"the LTS runs do not share one pass rate: {rates}") + passed, total = next(iter(rates.values())) + return f"{passed}/{total}" + + def check_pre_commit_commands(root: Path, guide: str) -> list[str]: """Compare the documented gate with the executable hook, in order.""" hook = root / ".githooks/pre-commit" @@ -869,11 +1187,14 @@ def check_framework_commands(root: Path) -> list[str]: def main() -> int: root = Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve() try: - texts = read_claim_files(root) + raw_texts = read_claim_files(root) + # The semantic checks read what a visitor reads: markers unwrapped. + texts = {rel: unwrap_markers(text) for rel, text in raw_texts.items()} baseline = load_test_baseline(root) problems = [] + problems += check_unmarked_figures(raw_texts) problems += check_pre_commit_commands(root, texts["docs/developer-guide.md"]) problems += check_framework_commands(root) problems += check_figure(texts, "Rust tests", baseline["tests"]) @@ -894,12 +1215,12 @@ def main() -> int: problems += check_validated_tiers(texts, root) problems += check_action_figures(texts, count_actions(root), count_action_specs(root)) - expected_tests = f"{baseline['tests']:,} Rust tests" for rel in REQUIRE_TEST_COUNT: - if rel in texts and expected_tests not in texts[rel]: + names = [m.name for m in find_markers(raw_texts[rel])[0]] + if "rust_tests" not in names: problems.append( - f"{rel}: does not state the measured baseline " - f"({expected_tests})" + f"{rel}: does not state the measured baseline; it needs a " + " marker" ) if problems: diff --git a/scripts/check_public_claims.sh b/scripts/check_public_claims.sh index 615ee010..4848b93a 100755 --- a/scripts/check_public_claims.sh +++ b/scripts/check_public_claims.sh @@ -61,6 +61,14 @@ if ! python3 "$script_dir/check_evidence_claims.py" "$repo_root"; then printf 'Invalid public claim: a published figure does not derive from evidence\n' >&2 exit 1 fi +# Every figure marker must render from its source. check_evidence_claims.py has +# already refused a figure outside a marker; this refuses a marker whose value +# is stale, unknown, malformed or inside a code fence. Run on the same tree, with +# no pipe, so its exit status is the one tested here. +if ! python3 "$script_dir/render_figures.py" --check "$repo_root"; then + printf 'Invalid public claim: a figure marker does not render from its source\n' >&2 + exit 1 +fi reject_pattern 'until npm publish lands|publish[- ]pending' \ 'setup package is documented as unpublished' "${claim_files[@]}" # SECURITY.md once said every enforcement layer is independent and a bypass of diff --git a/scripts/render_figures.py b/scripts/render_figures.py new file mode 100644 index 00000000..1afa6a93 --- /dev/null +++ b/scripts/render_figures.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +"""Render the figures in SysKnife's README and docs from their sources. + +A figure that describes the project is never typed. It sits in a marker: + + VALUE exact, thousands separators: 2,004 + VALUE rounded floor: 2,000+ + +and this script writes VALUE. HTML comments are invisible on GitHub and in mdBook. +`floor=K` renders floor(v / K) * K with a trailing "+", and refuses when v < K +(a floor of zero says nothing). A figure a routine PR moves is a floor; a figure +only a deliberate act changes (a recorded story run) stays exact. + + render_figures.py [--check] [ROOT] exit 1 naming file:line, the stored value and + the rendered one, for a stale marker, an + unknown NAME, a malformed marker (unbalanced, + nested) and a marker inside a code fence + render_figures.py --write [ROOT] rewrite only the marker contents; a second + run changes no byte + +Every value comes from a reader imported out of check_evidence_claims.py, the one +place that knows how to read each source: one source per figure, never a second +implementation here. A source that cannot be read, or a run that finds no marker +at all in CLAIM_FILES, is an error and exits 2: nothing is skipped. + +`UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh` records the test evidence and then +runs --write, so one command updates evidence and docs. +""" + +from __future__ import annotations + +import sys +from pathlib import Path +from typing import Callable + +SCRIPT_DIR = Path(__file__).resolve().parent +if str(SCRIPT_DIR) not in sys.path: + sys.path.insert(0, str(SCRIPT_DIR)) + +import check_evidence_claims as sources # noqa: E402 +from check_evidence_claims import Failure, find_markers # noqa: E402 + +# NAME -> reader. A NAME with an argument is `NAME:ARG` and its reader takes it. +SIMPLE: dict[str, Callable[[Path], "int | str"]] = { + "rust_tests": lambda root: sources.load_test_baseline(root)["tests"], + "frontend_tests": lambda root: sources.load_test_baseline(root)["frontend_tests"], + "typed_actions": sources.count_actions, + "action_specs": sources.count_action_specs, + "stories_total": lambda root: sum(sources.story_family_sizes(root).values()), + "stories_atomic": lambda root: sources.story_family_sizes(root)["atomic"], + "stories_ubuntu": lambda root: sources.story_family_sizes(root)["ubuntu"], + "uncovered_all": lambda root: sources.uncovered_action_counts(root)["All"], + "uncovered_fedora": lambda root: sources.uncovered_action_counts(root)["Fedora"], + "uncovered_ubuntu": lambda root: sources.uncovered_action_counts(root)["Ubuntu"], + "lts_story_total": sources.lts_story_total, + "lts_story_rate": sources.lts_story_rate, + "stories_ubuntu_unrecorded": lambda root: ( + sources.story_family_sizes(root)["ubuntu"] - sources.lts_story_total(root) + ), +} +PARAMETRIC: dict[str, Callable[[Path, str], "int | str"]] = { + "story_run": sources.story_run_rate, + "story_run_total": sources.story_run_total, + "story_run_served": sources.story_run_served, +} + + +def known_names() -> list[str]: + return sorted(SIMPLE) + [f"{name}:ARG" for name in sorted(PARAMETRIC)] + + +def read_value(root: Path, name: str, cache: dict) -> "int | str": + """Read one figure's source. Raises KeyError for an unknown NAME and Failure + when the source cannot be read.""" + if name in cache: + return cache[name] + if name in SIMPLE: + value = SIMPLE[name](root) + else: + base, _, arg = name.partition(":") + if base not in PARAMETRIC or not arg: + raise KeyError(name) + value = PARAMETRIC[base](root, arg) + cache[name] = value + return value + + +def render(value: "int | str", floor: "int | None") -> str: + """Format a source value. Raises ValueError when a floor cannot be applied.""" + if isinstance(value, bool) or not isinstance(value, (int, str)): + raise ValueError(f"source returned {value!r}") + if floor is None: + return f"{value:,}" if isinstance(value, int) else value + if not isinstance(value, int): + raise ValueError(f"floor={floor} needs a count, the source gave {value!r}") + if value < floor: + raise ValueError(f"value {value:,} is below floor={floor:,}; state it exactly or lower K") + return f"{value // floor * floor:,}+" + + +def process(root: Path, write: bool) -> tuple[list[str], list[str], int, int]: + """Check (or rewrite) every marker in CLAIM_FILES. + + Returns (problems, stale, marker count, files rewritten). A problem is what + --write cannot fix (unknown NAME, malformed marker, a floor that cannot apply, + no markers at all); stale markers are what it fixes. Nothing is written while + there is a problem. + """ + texts = sources.read_claim_files(root) + cache: dict = {} + problems: list[str] = [] + stale: list[str] = [] + rewrites: dict[str, str] = {} + total = 0 + for rel, text in texts.items(): + markers, structural = find_markers(text) + problems += [f"{rel}:{line}: {message}" for line, message in structural] + total += len(markers) + pieces, last = [], 0 + for marker in markers: + try: + value = read_value(root, marker.name, cache) + except KeyError: + problems.append( + f"{rel}:{marker.line}: unknown figure NAME {marker.name!r}; " + f"known: {', '.join(known_names())}" + ) + continue + try: + rendered = render(value, marker.floor) + except ValueError as exc: + problems.append(f"{rel}:{marker.line}: figure {marker.name!r}: {exc}") + continue + if rendered != marker.value: + stale.append( + f"{rel}:{marker.line}: stale figure {marker.name!r}: stored " + f"{marker.value!r}, rendered {rendered!r}" + ) + pieces.append(text[last : marker.value_start]) + pieces.append(rendered) + last = marker.value_end + pieces.append(text[last:]) + new_text = "".join(pieces) + if new_text != text: + rewrites[rel] = new_text + if total == 0: + problems.append("no figure markers found in CLAIM_FILES; refusing to treat that as success") + written = 0 + if write and not problems: + for rel, new_text in rewrites.items(): + with open(root / rel, "w", encoding="utf-8", newline="") as handle: + handle.write(new_text) + written += 1 + return problems, stale, total, written + + +def main(argv: list[str]) -> int: + args = [a for a in argv if not a.startswith("--")] + flags = {a for a in argv if a.startswith("--")} + if flags - {"--check", "--write"} or {"--check", "--write"} <= flags or len(args) > 1: + print("usage: render_figures.py [--check | --write] [ROOT]", file=sys.stderr) + return 2 + write = "--write" in flags + root = Path(args[0] if args else SCRIPT_DIR.parent).resolve() + try: + problems, stale, total, written = process(root, write) + except Failure as exc: + print(f"Figures could not be rendered: {exc}", file=sys.stderr) + return 2 + except (OSError, UnicodeDecodeError) as exc: + print(f"Figures could not be rendered: could not read or write a file: {exc}", file=sys.stderr) + return 2 + if problems or (stale and not write): + print("Figure markers that do not render from their sources:", file=sys.stderr) + for problem in problems + stale: + print(f" - {problem}", file=sys.stderr) + if write: + print("Nothing was written.", file=sys.stderr) + return 1 + if write: + print(f"Rendered {total} figure markers; rewrote {written} files.") + else: + print(f"All {total} figure markers match their sources.") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/tests/release/public-claims.test.sh b/tests/release/public-claims.test.sh index fa528a05..97d50e6a 100755 --- a/tests/release/public-claims.test.sh +++ b/tests/release/public-claims.test.sh @@ -173,6 +173,55 @@ if ! "$checker" "$fixture" >/dev/null 2>&1; then exit 1 fi + +# What render_figures.py renders for one figure out of the REAL tree, so the +# assertions below never retype a figure or its rounding. +rendered() { + python3 - "$repo_root" "$@" <<'PYEOF' +import sys +from pathlib import Path +sys.path.insert(0, sys.argv[1] + "/scripts") +import render_figures +root, name = Path(sys.argv[1]), sys.argv[2] +floor = int(sys.argv[3]) if len(sys.argv) > 3 else None +print(render_figures.render(render_figures.read_value(root, name, {}), floor)) +PYEOF +} + +# Replace the VALUE of the first marker NAME in a file and print the old value. +# Exits non-zero when no such marker exists, so a mutation cannot silently not +# apply. +set_marker_value() { + python3 - "$1" "$2" "$3" <<'PYEOF' +import re +import sys +from pathlib import Path +path, name, new = Path(sys.argv[1]), sys.argv[2], sys.argv[3] +text = path.read_text(encoding="utf-8") +pattern = re.compile( + r"()([^<\n]*)()" +) +match = pattern.search(text) +if not match: + raise SystemExit(f"no marker {name} in {path}") +path.write_text(text[:match.start(2)] + new + text[match.end(2):], encoding="utf-8") +print(match.group(2)) +PYEOF +} + +# Put one named claim file back from the real tree and prove the copy is exact: +# a restore that is not byte-identical would contaminate the next mutation. +restore_claim_file() { + local rel="$1" want got + cp "$repo_root/$rel" "$fixture/$rel" + want="$(sha256sum < "$repo_root/$rel")" + got="$(sha256sum < "$fixture/$rel")" + if [[ "$want" != "$got" ]]; then + printf 'FAIL: restore of %s is not byte-identical\n' "$rel" >&2 + exit 1 + fi +} + assert_rejected_with_diagnostic() { local label="$1" shift @@ -247,9 +296,10 @@ from pathlib import Path path = Path(sys.argv[1]) text = path.read_text(encoding="utf-8") +# The figure sits inside a marker; mutate the number, not the comment. pattern = re.compile( - r"(of the action names available on both families,\s*)(\d+)" - r"(\s+are still untouched by any story)" + r"(of the action names available on both families,\s*(?:)?)(\d+)" + r"((?:)?\s+are still untouched by any story)" ) match = pattern.search(text) if not match: @@ -266,7 +316,7 @@ if [[ -z "${derived_all:-}" || -z "${published_all:-}" ]]; then printf 'FAIL: story-coverage mutation produced no values\n' >&2 exit 1 fi -if ! grep -Fq "$published_all are still untouched by any story" \ +if ! grep -Fq "${published_all} are still untouched by any story" \ "$fixture/CONTRIBUTING.md"; then printf 'FAIL: story-coverage mutation did not apply\n' >&2 exit 1 @@ -707,31 +757,33 @@ assert_rejected() { fi } -# Derive the baseline from the evidence artifact rather than repeating the -# literal here. When the baseline moved, this mutation silently became a no-op: -# the sed matched nothing, the checker legitimately passed, and the assertion -# below then blamed the checker. A test whose mutation can stop mutating is worse -# than no test — so every mutation below is confirmed to have applied. +# A typed test count is refused twice over: as a figure outside a marker, and, +# for an exact figure, because it disagrees with the evidence artifact. Derive the +# wrong value from the artifact rather than repeating a literal; a test whose +# mutation can stop mutating is worse than no test, so every mutation below is +# confirmed to have applied. baseline="$(python3 -c " import json,sys -print(f\"{json.load(open(sys.argv[1]))['tests']:,}\") +print(f\"{json.load(open(sys.argv[1]))['tests'] + 1:,}\") " "$repo_root/tests/evidence/workspace-tests.json")" if [[ -z "$baseline" ]]; then printf 'FAIL: could not read the test count from the evidence artifact\n' >&2 exit 1 fi -sed -i "s/${baseline} Rust tests/1,256 Rust tests/" "$fixture/README.md" -if ! grep -q '1,256 Rust tests' "$fixture/README.md"; then +printf '\n%s Rust tests are typed here by hand.\n' "$baseline" >> "$fixture/README.md" +if ! grep -Fq "$baseline Rust tests are typed here" "$fixture/README.md"; then printf 'FAIL: baseline mutation did not apply; fixture would pass vacuously\n' >&2 exit 1 fi -assert_rejected 'test count that disagrees with the evidence artifact' +assert_rejected_with_diagnostic 'test count that disagrees with the evidence artifact' \ + "README.md:" "claims $baseline Rust tests, evidence says" "is not inside a" cp "$repo_root/README.md" "$fixture/README.md" # A figure with no artifact at all behind it. This is the "65/65 stories" shape: # published for months, never produced by a run. printf '\nUbuntu 24.04 is validated with 47/50 stories on a live VM.\n' >> "$fixture/README.md" -assert_rejected 'story pass rate with no recorded run' +assert_rejected_with_diagnostic 'story pass rate with no recorded run' \ + 'claims 47/50 stories with no run' 'to back it' cp "$repo_root/README.md" "$fixture/README.md" # ...and the same claim IS accepted once a run records it. Without this the rule @@ -757,13 +809,20 @@ write_fixture_run() { EV_SKIPPED=0 EV_RATELIMITED=0 \ python3 "$repo_root/scripts/record_story_run.py" } +# A story pass rate is claimed inside a marker, so the claim reaches the semantic +# rules below with nothing else to reject it: the marker reads the run's own +# totals, and only "does a full-family run back this" stands between the claim and +# the checker. `fixture_claim` writes the claim line for the fixture run. +fixture_claim() { + printf '\nUbuntu 24.04 is validated with %s stories on a live VM.\n' "$1" +} write_fixture_run ubuntu 50 47 ok # The writer's output must be valid JSON — the defect it replaced was not. python3 -m json.tool "$fixture/tests/evidence/story-runs/fixture-run.json" >/dev/null \ || { printf 'FAIL: record_story_run.py emitted invalid JSON\n' >&2; exit 1; } -printf '\nUbuntu 24.04 is validated with 47/50 stories on a live VM.\n' >> "$fixture/README.md" -if ! "$checker" "$fixture" >/dev/null 2>&1; then - printf 'FAIL: checker rejected a story claim that its recorded run backs\n' >&2 +fixture_claim 47/50 >> "$fixture/README.md" +if ! output="$("$checker" "$fixture" 2>&1)"; then + printf 'FAIL: checker rejected a story claim that its recorded run backs\n%s\n' "$output" >&2 exit 1 fi @@ -783,7 +842,8 @@ grep -q '"skipped": 1' "$fixture/tests/evidence/story-runs/fixture-run.json" || printf 'FAIL: skipped-story mutation did not apply\n' >&2 exit 1 } -assert_rejected 'story pass rate backed by skipped stories' +assert_rejected_with_diagnostic 'story pass rate backed by skipped stories' \ + 'claims 47/50 stories with no run' 'to back it' write_fixture_run ubuntu 50 47 ok # Rate-limited stories are a separate diagnostic, but they are just as unable @@ -801,30 +861,34 @@ grep -q '"rate_limited": 1' "$fixture/tests/evidence/story-runs/fixture-run.json printf 'FAIL: rate-limited-story mutation did not apply\n' >&2 exit 1 } -assert_rejected 'story pass rate backed by rate-limited stories' +assert_rejected_with_diagnostic 'story pass rate backed by rate-limited stories' \ + 'claims 47/50 stories with no run' 'to back it' write_fixture_run ubuntu 50 47 ok # Backed by a run, but attributed to a model that did not produce it. -sed -i 's|47/50 stories on a live VM.|47/50 stories on a live VM with gpt-4.1.|' "$fixture/README.md" +sed -i 's|47/50 stories on a live VM.|47/50 stories on a live VM with gpt-4.1.|' "$fixture/README.md" grep -q 'gpt-4.1' "$fixture/README.md" || { printf 'FAIL: model-attribution mutation did not apply\n' >&2 exit 1 } -assert_rejected 'story pass rate attributed to the wrong model' +assert_rejected_with_diagnostic 'story pass rate attributed to the wrong model' \ + 'attributes 47/50 stories to gpt-4.1' cp "$repo_root/README.md" "$fixture/README.md" # A replay the harness declared unproven must not back a figure either. The # artifact used to be written before the cassette audit ran, so a run that missed # every call still produced a file with a healthy pass rate. write_fixture_run ubuntu 50 47 failed -printf '\nUbuntu 24.04 is validated with 47/50 stories on a live VM.\n' >> "$fixture/README.md" -assert_rejected 'story pass rate from a replay whose cassette audit failed' +fixture_claim 47/50 >> "$fixture/README.md" +assert_rejected_with_diagnostic 'story pass rate from a replay whose cassette audit failed' \ + 'claims 47/50 stories with no run' 'to back it' cp "$repo_root/README.md" "$fixture/README.md" # A subset run must never back a headline: probes record an empty story_set. write_fixture_run "" 4 4 ok -printf '\nUbuntu 24.04 is validated with 4/4 stories on a live VM.\n' >> "$fixture/README.md" -assert_rejected 'four-story probe quoted as a headline figure' +fixture_claim 4/4 >> "$fixture/README.md" +assert_rejected_with_diagnostic 'four-story probe quoted as a headline figure' \ + 'claims 4/4 stories with no run' 'to back it' cp "$repo_root/README.md" "$fixture/README.md" rm -f "$fixture/tests/evidence/story-runs/fixture-run.json" @@ -834,32 +898,21 @@ assert_rejected 'a claim file missing from the checked surface' mv "$fixture/ROADMAP.held" "$fixture/ROADMAP.md" # The action count is derived from the catalogue source, so a retyped one fails. -actions="$(grep -oE '[0-9]+ typed actions' "$fixture/docs/introduction.md" | head -1 | grep -oE '^[0-9]+' || true)" -if [[ -z "$actions" ]]; then - printf 'FAIL: could not find the typed-action count in the fixture\n' >&2 +printf '\nThe catalogue holds 999 typed actions.\n' >> "$fixture/docs/introduction.md" +if ! grep -Fq '999 typed actions' "$fixture/docs/introduction.md"; then + printf 'FAIL: action-count mutation did not apply\n' >&2 exit 1 fi -sed -i "s/${actions} typed actions/999 typed actions/" "$fixture/docs/introduction.md" -assert_rejected 'action count that disagrees with the catalogue source' +assert_rejected_with_diagnostic 'action count that disagrees with the catalogue source' \ + 'docs/introduction.md' 'claims 999 typed actions, evidence says' cp "$repo_root/docs/introduction.md" "$fixture/docs/introduction.md" # An ActionSpec qualifier must not exempt a stale subset count from evidence. -python3 - "$fixture/docs/developer-guide.md" <<'PYEOF' -import re -import sys -from pathlib import Path -path = Path(sys.argv[1]) -text = path.read_text(encoding="utf-8") -updated, count = re.subn( - r"[0-9]+ actions with an `ActionSpec`", - "4 actions with an `ActionSpec`", - text, - count=1, -) -if count != 1: - raise SystemExit("could not find the ActionSpec count in the fixture") -path.write_text(updated, encoding="utf-8") -PYEOF +printf '\nThe executor holds 4 actions with an `ActionSpec`.\n' >> "$fixture/docs/developer-guide.md" +if ! grep -Fq '4 actions with an `ActionSpec`' "$fixture/docs/developer-guide.md"; then + printf 'FAIL: ActionSpec mutation did not apply\n' >&2 + exit 1 +fi assert_rejected_with_diagnostic \ 'ActionSpec count that disagrees with the generated table' \ 'developer-guide.md' 'claims 4 actions with an ActionSpec' 'derived' @@ -939,9 +992,10 @@ cp "$repo_root/docs/introduction.md" "$fixture/docs/introduction.md" # ...and the counterpart: a real family size must still be quotable, or the guard # would force every honest mention of the suite out of the docs. -printf '\nThe atomic family is 54 stories.\n' >> "$fixture/docs/introduction.md" -if ! "$checker" "$fixture" >/dev/null 2>&1; then - printf 'FAIL: checker rejected a real family size (54 stories)\n' >&2 +printf '\nThe atomic family is %s stories.\n' \ + "$(rendered stories_atomic 10)" >> "$fixture/docs/introduction.md" +if ! output="$("$checker" "$fixture" 2>&1)"; then + printf 'FAIL: checker rejected a real, marked family size\n%s\n' "$output" >&2 exit 1 fi cp "$repo_root/docs/introduction.md" "$fixture/docs/introduction.md" @@ -1029,4 +1083,5 @@ grep -q '"rate_limited": 1' \ } assert_rejected 'Validated tier backed by a replay with rate-limited stories' + printf 'Public claims contract passed.\n' From b8f7ba597a3bd6a1fc7d2d0674b8a33d72988a7b Mon Sep 17 00:00:00 2001 From: Vladimir Rotariu Date: Tue, 27 Oct 2026 01:56:05 -0600 Subject: [PATCH 2/3] build: re-render the docs when the test baseline is recorded UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh now runs render_figures.py --write after recording, so one command updates the evidence and every figure derived from it. A failed render fails the recording. The provenance test stubs the renderer and checks the call, the arguments, that a verifying run never renders, and that a render failure is not swallowed. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU --- scripts/test_baseline.sh | 8 +++++ .../release/test-baseline-provenance.test.sh | 30 +++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/scripts/test_baseline.sh b/scripts/test_baseline.sh index 10924f72..cb89ea1a 100755 --- a/scripts/test_baseline.sh +++ b/scripts/test_baseline.sh @@ -15,6 +15,10 @@ # UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh # Rust workspace # UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh --frontend # vitest # +# Either one then runs scripts/render_figures.py --write, which re-renders every +# figure marker in the README and docs from the evidence. Commit the evidence +# file and whatever it re-rendered. +# # Without UPDATE_TEST_BASELINE the run verifies instead: a suite whose size no # longer matches the recorded baseline fails, because a published claim derives # from that baseline. Remaining arguments are forwarded to the runner, so this is @@ -104,6 +108,10 @@ if [[ "${UPDATE_TEST_BASELINE:-0}" == "1" ]]; then --count "$count" printf '\ntest_baseline: recorded %s %s tests in %s\n' \ "$count" "$suite" "${artifact#"$repo_root"/}" + # The published figures derive from the evidence just recorded, so re-render + # them in the same step: one command updates evidence and docs. No pipe, so a + # failed render is this script's exit status. + python3 "$repo_root/scripts/render_figures.py" --write "$repo_root" exit 0 fi diff --git a/tests/release/test-baseline-provenance.test.sh b/tests/release/test-baseline-provenance.test.sh index f043af4b..5f1639c9 100644 --- a/tests/release/test-baseline-provenance.test.sh +++ b/tests/release/test-baseline-provenance.test.sh @@ -121,6 +121,19 @@ mkdir -p "$fake_bin" "$sandbox/scripts" "$sandbox/tests/evidence" \ "$sandbox/apps/sysknife-shell/node_modules/.bin" cp "$repo_root/scripts/test_baseline.sh" "$sandbox/scripts/test_baseline.sh" cp "$recorder" "$sandbox/scripts/record_test_baseline.py" +# The recording step re-renders the published figures. The real renderer needs +# the whole claim surface; the sandbox holds a stand-in that logs how it was +# called and can be made to fail, which is what the wiring has to be tested with. +render_log="$fixture/render-calls.log" +cat > "$sandbox/scripts/render_figures.py" <<'PYEOF' +import os +import sys +with open(os.environ["RENDER_LOG"], "a", encoding="utf-8") as log: + log.write(" ".join(sys.argv[1:]) + "\n") +sys.exit(int(os.environ.get("FAKE_RENDER_STATUS", "0"))) +PYEOF +export RENDER_LOG="$render_log" +: > "$render_log" cp "$repo_root/tests/evidence/workspace-tests.json" \ "$sandbox/tests/evidence/workspace-tests.json" python3 - "$fake_bin/cargo" "$fake_bin/vitest" <<'PY' @@ -210,7 +223,24 @@ if python3 "$recorder" --artifact "$legacy_read" --field tests --read >/dev/null fi cmp "$legacy_read_before" "$legacy_read" +# A verifying run renders nothing; only a recording run re-renders, with --write, +# and a renderer that fails fails the recording. +if [[ -s "$render_log" ]]; then + printf 'FAIL: a verifying run called the figure renderer: %s\n' "$(cat "$render_log")" >&2 + exit 1 +fi +if UPDATE_TEST_BASELINE=1 FAKE_RENDER_STATUS=3 FAKE_RUST_COUNT=$((rust_count + 1)) \ + run_baseline >/dev/null 2>&1; then + printf 'FAIL: a failed figure render was swallowed by the recording run\n' >&2 + exit 1 +fi +: > "$render_log" UPDATE_TEST_BASELINE=1 FAKE_RUST_COUNT=$((rust_count + 1)) run_baseline >/dev/null +if [[ "$(cat "$render_log")" != "--write $sandbox" ]]; then + printf 'FAIL: recording did not run render_figures.py --write on the tree: %s\n' \ + "$(cat "$render_log")" >&2 + exit 1 +fi python3 - "$sandbox/tests/evidence/workspace-tests.json" "$frontend_count" "$rust_count" <<'PY' import json import sys From e80b8af0cff9e82717448f7ef8593c9acf92cf98 Mon Sep 17 00:00:00 2001 From: Vladimir Rotariu Date: Tue, 27 Oct 2026 01:56:42 -0600 Subject: [PATCH 3/3] test: prove the figure renderer's refusals bite Adds to public-claims.test.sh: a stale marker (names file and both values), an unknown NAME, a figure outside a marker, a marker inside a code fence, an unbalanced and a nested marker, a required doc that stopped stating the test count, the floor boundary (v = K*n, v = K*n - 1, v = K, v < K), --write then --check, a --write that touches only marker contents, a second --write that changes no byte, an unreadable and a corrupt evidence file, and a tree with no markers. Each refusal was mutation-proved: the guard removed in a scratch copy, the test seen to go red, the file restored from a pristine copy and hash-checked. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01UpyrhbDTW1gBwzGAC44GYU --- tests/release/public-claims.test.sh | 297 ++++++++++++++++++++++++++++ 1 file changed, 297 insertions(+) diff --git a/tests/release/public-claims.test.sh b/tests/release/public-claims.test.sh index 97d50e6a..f2f452c1 100755 --- a/tests/release/public-claims.test.sh +++ b/tests/release/public-claims.test.sh @@ -1084,4 +1084,301 @@ grep -q '"rate_limited": 1' \ assert_rejected 'Validated tier backed by a replay with rate-limited stories' +# --------------------------------------------------------------------------- +# Figures are rendered, not typed (scripts/render_figures.py). +# +# Each refusal below is proved to bite by deleting its guard (see the mutation +# notes in the commit that added it); a test here that stays green with its +# guard removed is decorative. +# --------------------------------------------------------------------------- +renderer="$repo_root/scripts/render_figures.py" + +# The tier tests above left a mutated replay twin behind. Put every story run +# back, and prove the whole fixture passes again, before attacking figures: a +# rejection below must come from the figure under test. +cp "$repo_root"/tests/evidence/story-runs/*.json "$fixture/tests/evidence/story-runs/" +if ! output="$("$checker" "$fixture" 2>&1)"; then + printf 'FAIL: the fixture is rejected before the figure tests begin\n%s\n' "$output" >&2 + exit 1 +fi + +# A stale marker fails, names the file and line, and shows both values. Only the +# renderer can catch this one: the stored floor is not an exact count, so no +# semantic check compares it. +want_rust="$(rendered rust_tests 1000)" +old_value="$(set_marker_value "$fixture/README.md" rust_tests '1,000+')" +if [[ "$old_value" != "$want_rust" || "$want_rust" == '1,000+' ]]; then + printf 'FAIL: stale-marker mutation did not change the value (was %s, rendered %s)\n' \ + "$old_value" "$want_rust" >&2 + exit 1 +fi +assert_rejected_with_diagnostic 'a stale figure marker' \ + 'README.md:' "stale figure 'rust_tests'" "stored '1,000+'" "rendered '$want_rust'" +restore_claim_file README.md + +# An unknown NAME fails and lists what is known. +python3 - "$fixture/docs/architecture.md" <<'PYEOF' +import sys +from pathlib import Path +path = Path(sys.argv[1]) +text = path.read_text(encoding="utf-8") +if text.count("fig:typed_actions ") != 1: + raise SystemExit("expected exactly one typed_actions marker in architecture.md") +path.write_text(text.replace("fig:typed_actions ", "fig:typed_actionz "), encoding="utf-8") +PYEOF +grep -Fq 'fig:typed_actionz' "$fixture/docs/architecture.md" || { + printf 'FAIL: unknown-NAME mutation did not apply\n' >&2 + exit 1 +} +assert_rejected_with_diagnostic 'an unknown figure NAME' \ + 'docs/architecture.md:' "unknown figure NAME 'typed_actionz'" 'rust_tests' +restore_claim_file docs/architecture.md + +# A figure outside a marker fails, even a rounded floor that no semantic check +# could validate. This is the loophole: without it a hand-typed number creeps back. +printf '\nThe suite has 2,000+ Rust tests.\n' >> "$fixture/README.md" +assert_rejected_with_diagnostic 'a hand-typed floor outside a marker' \ + 'README.md:' "figure '2,000+ Rust tests' is not inside a" +restore_claim_file README.md +printf '\nThe catalogue holds 190+ typed actions and 72 frontend tests.\n' >> "$fixture/README.md" +assert_rejected_with_diagnostic 'hand-typed action and frontend figures' \ + "figure '190+ typed actions'" "figure '72 frontend tests'" +restore_claim_file README.md + +# The two README badges read the evidence file and carry no figure. They must be +# present, or the pristine pass above says nothing about them. +for badge in 'query=$.tests&label=Rust%20tests' 'query=$.frontend_tests&label=frontend%20tests'; do + if ! grep -Fq "$badge" "$fixture/README.md"; then + printf 'FAIL: README is missing the live badge %s\n' "$badge" >&2 + exit 1 + fi +done + +# A marker inside a code fence fails; one inside an inline code span is syntax +# documentation and does not. +printf '\n```sh\n# %s\n```\n' "$want_rust" >> "$fixture/README.md" +assert_rejected_with_diagnostic 'a figure marker inside a code fence' \ + 'README.md:' 'figure marker inside a fenced code block' +restore_claim_file README.md +printf '\nWrite `VALUE` around a figure.\n' >> "$fixture/README.md" +if ! output="$("$checker" "$fixture" 2>&1)"; then + printf 'FAIL: a marker shown in an inline code span was parsed\n%s\n' "$output" >&2 + exit 1 +fi +restore_claim_file README.md + +# Malformed markers: unbalanced and nested. +printf '\nSee %s for the size.\n' "$want_rust" >> "$fixture/README.md" +assert_rejected_with_diagnostic 'an unbalanced figure marker' \ + 'README.md:' 'unbalanced figure marker' +restore_claim_file README.md +printf '\nSee 1.\n' >> "$fixture/README.md" +assert_rejected_with_diagnostic 'a nested figure marker' \ + 'README.md:' 'nested figure marker' +restore_claim_file README.md + +# A doc that must state the test count loses its marker: dropping the figure +# must not pass silently. +python3 - "$fixture/docs/distro-support.md" <<'PYEOF' +import re +import sys +from pathlib import Path +path = Path(sys.argv[1]) +text = path.read_text(encoding="utf-8") +updated, count = re.subn(r"[^<]* Rust tests", "unit tests", text) +if count != 1: + raise SystemExit("expected one rust_tests marker in distro-support.md") +path.write_text(updated, encoding="utf-8") +PYEOF +assert_rejected_with_diagnostic 'a required test-count doc that stopped stating it' \ + 'docs/distro-support.md' 'does not state the measured baseline' +restore_claim_file docs/distro-support.md + +# Rounding at the boundary, against fixture evidence: v = K*n renders n*K+, and +# v = K*n - 1 renders (n-1)*K+, and v < K refuses. The fixture evidence is changed +# through the real recorder's schema (only the `tests` field moves). +set_tests() { + python3 - "$fixture/tests/evidence/workspace-tests.json" "$1" <<'PYEOF' +import json +import sys +path = sys.argv[1] +doc = json.loads(open(path, encoding="utf-8").read()) +doc["tests"] = int(sys.argv[2]) +open(path, "w", encoding="utf-8").write(json.dumps(doc, indent=2, sort_keys=True) + "\n") +PYEOF +} +marker_of() { # file name -> the first marker's stored value + python3 - "$1" "$2" <<'PYEOF' +import re +import sys +text = open(sys.argv[1], encoding="utf-8").read() +match = re.search(r"([^<\n]*)