diff --git a/CHANGELOG.md b/CHANGELOG.md
index 013de158..a846fc3f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,21 @@ Releases before `0.2.5` predate the public launch; their notes live in the
## [Unreleased]
+### Changed
+
+- **Published figures are rendered from their sources, never typed.**
+ `scripts/render_figures.py` rewrites every `` marker in the
+ README and docs (test counts, action counts, story-set sizes, recorded pass
+ rates), and `scripts/check_evidence_claims.py` now refuses a figure outside a
+ marker. Figures that routine PRs move are rounded floors ("2,000+"); recorded
+ story pass rates stay exact. `UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh`
+ records the evidence and re-renders the docs in one step, and
+ `scripts/check_public_claims.sh` runs `render_figures.py --check`. The README
+ shows the exact Rust and frontend test counts as live badges read from
+ `tests/evidence/workspace-tests.json`. The README line that said the Ubuntu
+ suite "has since grown to 96" was already stale (it holds 99) and is now a
+ derived figure.
+
### Security
- **`request_hash` is now keyed: an audit export is no longer an offline
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index bf6cada9..06e1b0a0 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -65,10 +65,10 @@ table below.
| Area | Why it matters | Difficulty |
|---|---|---|
-| **Ubuntu LTS support** | The current suite is 99 Ubuntu stories. Committed live-VM evidence covers 79 of those Ubuntu stories on each LTS release, with a committed replay twin that reproduces each run: 22.04, 24.04 and 26.04 all at 79/79. The twenty additional stories are not yet included in a committed live-VM run. `ubuntu-vm.sh` accepts `UBUNTU_RELEASE=jammy\|noble\|resolute`. Remaining: story coverage for the cross-family actions, and every Debian-only action has a story. | medium |
+| **Ubuntu LTS support** | The current suite is 90+ Ubuntu stories. Committed live-VM evidence covers 79 of those Ubuntu stories on each LTS release, with a committed replay twin that reproduces each run: 22.04, 24.04 and 26.04 all at 79/79. The 20+ additional stories are not yet included in a committed live-VM run. `ubuntu-vm.sh` accepts `UBUNTU_RELEASE=jammy\|noble\|resolute`. Remaining: story coverage for the cross-family actions, and every Debian-only action has a story. | medium |
| **Distro detection coverage** | Robust `/etc/os-release` parsing for every release we claim to support. Pure-function tests against real fixture files, no integration mocks. The existing fixtures at the bottom of `crates/sysknife-core/src/distro.rs` show the shape. | easy |
| **Action catalogue gaps** | Add a typed action (for example `EnableFirewallZone`). Small and isolated, and every PR carries the policy entry, the risk level and the tests. | easy |
-| **E2E story coverage** | Real prompts, real LLM, real daemon. The suite is 153 stories: 54 atomic + 99 Ubuntu. What is left is the cross-family middle: of the action names available on both families, 46 are still untouched by any story, plus 10 Fedora-only and 0 Ubuntu-only ones. See #233 for the clustered map. | medium |
+| **E2E story coverage** | Real prompts, real LLM, real daemon. The suite is 150+ stories: 50+ atomic + 90+ Ubuntu. What is left is the cross-family middle: of the action names available on both families, 46 are still untouched by any story, plus 10 Fedora-only and 0 Ubuntu-only ones. See #233 for the clustered map. | medium |
| **Fedora Atomic validation** | The action families exist and `DistroId::is_supported()` returns true for Atomic 41 and up. Nobody has run `tests/e2e/atomic-vm.sh` against a current release. Needs Fedora Atomic hardware or a VM host. | tedious |
| **Demo recording on real hardware** | Replace the bundled demo GIF with a 30-second recording on real Ubuntu 26.04 with rollback visible. | easy |
@@ -168,16 +168,21 @@ manifests. A per-crate run like
`cargo test -p sysknife-types` never touches those, so a green crate is not a
green board.
-**Adding or removing a Rust test moves a published figure.** The suite size lives
-in `tests/evidence/workspace-tests.json`, and `README.md`,
-`docs/introduction.md` and `docs/distro-support.md` state it in prose. Regenerate
-the artifact and update those three files in the same commit:
+**Never type a figure.** Test counts, the action count, story-set sizes and
+story pass rates are rendered by `scripts/render_figures.py` from the artifact or
+source that produces them, and a number the claim checks match outside a marker
+fails the build. Add your tests, run the command below, and commit the evidence
+file together with whatever it re-rendered:
```sh
UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh
-grep -rn 'Rust tests' README.md docs/introduction.md docs/distro-support.md
```
+Figures that routine pull requests move are rounded floors ("2,000+"), so most
+PRs re-render nothing. The exact Rust and frontend counts are the README badges,
+read from the evidence file. The marker syntax is explained once, in
+[docs/contributing/testing.md](docs/contributing/testing.md#published-figures).
+
`workspace-tests.json` stores the expected count for each suite and the canonical
command that measures it. Verification re-runs the suite on the current
checkout and requires the observed count to match. It makes no historical claim
diff --git a/HACKING.md b/HACKING.md
index d4502133..77e15b84 100644
--- a/HACKING.md
+++ b/HACKING.md
@@ -631,12 +631,18 @@ chained to the previous one.
short reference version of this file.
- [`docs/testing/user-stories.md`](docs/testing/user-stories.md) — the
20 worked Atomic scenarios and their pass criteria (the atomic family itself
- is 54 stories; the Ubuntu family is a further 99).
+ is 50+ stories; the Ubuntu family is a further 90+).
- [`tests/e2e/atomic-vm.sh help`](tests/e2e/atomic-vm.sh) — the
subcommand reference, kept in sync with the script itself.
## 17. Adding a new action
+Published figures (the action count, test counts, story-set sizes) are never typed:
+they are `fig` markers rendered by `scripts/render_figures.py`. After adding an
+action or a story, run `UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh` and commit
+the evidence file and whatever it re-rendered. Syntax:
+[docs/contributing/testing.md](docs/contributing/testing.md#published-figures).
+
An "action" is a named operation the daemon can perform — `GetDiskUsage`,
`AddLayeredPackage`, `CreateUser`, etc. Action names are string literals
duplicated across several independent match expressions and lists. The
diff --git a/README.md b/README.md
index 2718ef18..dfb41554 100644
--- a/README.md
+++ b/README.md
@@ -18,6 +18,8 @@
+ 
+ 
@@ -234,9 +236,9 @@ privilege boundary is worse than none:
sudo make uninstall
```
-All three Ubuntu LTS releases record a live-VM run of 79 stories from the Ubuntu
-suite (which has since grown to 96), and each run has a replay twin that reproduces it: 22.04, 24.04 and 26.04
-all at 79/79, every twin serving every call with zero misses. The runs are in
+All three Ubuntu LTS releases record a live-VM run of 79 stories from the Ubuntu
+suite (which has since grown to 90+), and each run has a replay twin that reproduces it: 22.04, 24.04 and 26.04
+all at 79/79, every twin serving every call with zero misses. The runs are in
`tests/evidence/story-runs/`. The suite grew from 50 when every Debian-only
action got a story, `GetHostState` first.
Fedora Atomic is the rpm-ostree target; `make install` does not complete on
@@ -342,18 +344,18 @@ milestone.
| Component | State |
|---|---|
| `sysknife-brain` — LLM planner, tool loop, safety fence | ✅ |
-| `sysknife-daemon` — 192 typed actions, auth, preview, transactions | ✅ |
+| `sysknife-daemon` — 190+ typed actions, auth, preview, transactions | ✅ |
| Live IPC + streaming + atomic-host rollback (rpm-ostree) | ✅ |
| Terminal approval gate — one-time, TTL-bounded receipts | ✅ |
| MCP server (Claude Code / Cursor / any MCP client) | ✅ |
| Tamper-evident Ed25519-signed audit chain | ✅ |
| RFC 5424 syslog forwarding (Splunk / Sentinel / QRadar) | ✅ |
| Postgres backend (RDS / Cloud SQL / Neon / Supabase) | ✅ |
-| **Ubuntu support** — 79/79 stories on a live 22.04 VM, recorded in `tests/evidence/story-runs/` | ✅ |
-| **Every Ubuntu LTS validated** — 22.04, 24.04 and 26.04 all at 79/79, each with a replay twin that reproduces it | ✅ |
+| **Ubuntu support** — 79/79 stories on a live 22.04 VM, recorded in `tests/evidence/story-runs/` | ✅ |
+| **Every Ubuntu LTS validated** — 22.04, 24.04 and 26.04 all at 79/79, each with a replay twin that reproduces it | ✅ |
| Telegram approval interface | 📋 roadmap |
-**2,011 Rust tests and 72 frontend tests** form the current deterministic
+**2,000+ Rust tests and 70+ frontend tests** form the current deterministic
release baseline.
## Configure your LLM
@@ -399,8 +401,8 @@ All config files that may contain API keys are created with `chmod 0600`.
See [ROADMAP.md](ROADMAP.md) for the full milestone breakdown.
-- ✅ **Ubuntu 22.04** — 79/79 stories on a live VM (recorded in `tests/evidence/story-runs/`)
-- ✅ **Ubuntu 24.04 and 26.04** — 79/79 and 79/79 on live VMs; every LTS run has a replay twin that reproduces it
+- ✅ **Ubuntu 22.04** — 79/79 stories on a live VM (recorded in `tests/evidence/story-runs/`)
+- ✅ **Ubuntu 24.04 and 26.04** — 79/79 and 79/79 on live VMs; every LTS run has a replay twin that reproduces it
- ✅ `sysknife audit export` — stored signed chain rows as JSON with `--since` / `--limit`
- 📋 Telegram inline-button approvals
- 📋 CEF / NDJSON output modes for SIEM ingest
diff --git a/docs/architecture.md b/docs/architecture.md
index e89c55ed..c1ead602 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -72,7 +72,7 @@ component uses the same resolution order.
Privileged. The only component that touches the system. Provides:
-- 192 typed actions (rpm-ostree, systemd, firewall, users, containers,
+- 190+ typed actions (rpm-ostree, systemd, firewall, users, containers,
flatpak, toolbox, SSH, kernel args, …)
- Role-based authorization (`Observer` → `Dev` → `Admin`, plus `Boot`)
- Policy enforcement: stale-approval detection, request hash validation
diff --git a/docs/contributing/testing.md b/docs/contributing/testing.md
index c4ae80ec..6df14fe7 100644
--- a/docs/contributing/testing.md
+++ b/docs/contributing/testing.md
@@ -13,7 +13,7 @@ validation before a release.
| Integration (Rust) | Daemon IPC, safety fence, policy | <10s | Every commit, every CI run |
| **Dev stories (local, no VM)** | LLM plan structure for read-only stories; runs on any Linux host | 1-3 min | After brain/prompt changes |
| CI smoke (container) | Daemon + Ollama + read-only stories in a Linux runner | 5-10 min | Opt-in (manual `workflow_dispatch` only) |
-| E2E Atomic VM | **Real Silverblue** in QEMU/KVM, full stack, all 54 stories | 15-30 min first boot; 2-3 min subsequent | Local / pre-release |
+| E2E Atomic VM | **Real Silverblue** in QEMU/KVM, full stack, all 50+ stories | 15-30 min first boot; 2-3 min subsequent | Local / pre-release |
| **E2E Ubuntu VM** | **Ubuntu 24.04** cloud image, full story suite (pass rate in `tests/evidence/story-runs/`) | ~15 min first boot; ~2 min subsequent | After Ubuntu action changes |
| Manual QA | Real Silverblue/Kinoite hardware, destructive actions, GUI | 30-60 min | Before releases + demo video |
@@ -275,7 +275,7 @@ sudo chmod +r /boot/vmlinuz-*
# Run the read-only stories (non-destructive default)
./tests/e2e/atomic-vm.sh run
-# Run ALL 54 stories including destructive — restore the baseline afterwards.
+# Run ALL stories including destructive — restore the baseline afterwards.
SYSKNIFE_ALLOW_DESTRUCTIVE=1 ./tests/e2e/atomic-vm.sh run
# Roll back to the clean baseline so the next run is fast
@@ -376,12 +376,43 @@ sudo -E tests/e2e/run-stories.sh 1 4 7
Per-story logs are written to `tests/e2e/logs/story-N.log`.
+## Published figures
+
+Never type a figure. A count the README or the docs state about the project
+(tests, actions, story-set sizes, recorded pass rates) is rendered from its
+source by `scripts/render_figures.py`, and `scripts/check_evidence_claims.py`
+refuses one that sits outside a marker. After you add tests, run
+`UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh` (add `--frontend` for the
+vitest suite), then commit the evidence file and whatever it re-rendered.
+
+A marker is an HTML comment pair around the value, invisible on GitHub and in
+mdBook:
+
+- `VALUE` renders the exact value with thousands
+ separators. Story pass rates read from a recorded run are exact.
+- `VALUE` renders the value rounded down to
+ a multiple of `K` with a trailing plus. A figure a routine PR moves is a
+ floor, so most PRs change no prose. `render_figures.py` refuses a floor above
+ the value.
+
+`NAME` is a key in the registry at the top of `scripts/render_figures.py`; each
+one reads its source through a function from `check_evidence_claims.py`. A
+marker sits on one line, cannot nest, and cannot sit inside a fenced code
+block. Inline code spans are ignored, which is how this section shows the
+syntax. `scripts/render_figures.py --check` names the file and line of every
+stale, unknown or malformed marker, and `--write` fixes the stale ones.
+
+The exact Rust and frontend test counts are the two README badges, which read
+`tests/evidence/workspace-tests.json` from `main`.
+
## Before opening a PR
1. `cargo nextest run --workspace && npm test --prefix apps/sysknife-shell` — required, fast
2. `cargo clippy --workspace --all-features --all-targets --locked -- -D warnings`
3. `cargo fmt --all --check`
-4. For changes to the brain, daemon, IPC, or action catalogue:
+4. If you added or removed tests, `UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh`
+ and commit what it changed (see [Published figures](#published-figures))
+5. For changes to the brain, daemon, IPC, or action catalogue:
- Run the VM tests locally (`atomic-vm.sh` flow)
- Optionally run the CI smoke test by dispatching the `e2e` workflow
@@ -393,7 +424,7 @@ The maintainer runs these in order:
2. VM tests (at least Silverblue + one other atomic variant) pass locally
3. Manual QA on real Silverblue hardware using
[docs/testing/user-stories.md](../testing/user-stories.md) as the
- checklist — all 54 stories including destructive ones
+ checklist — all 50+ stories including destructive ones
4. Record the demo video on real hardware (issue #32)
## Troubleshooting
diff --git a/docs/contributing/ubuntu-vm-testing.md b/docs/contributing/ubuntu-vm-testing.md
index 328da748..2f1b370d 100644
--- a/docs/contributing/ubuntu-vm-testing.md
+++ b/docs/contributing/ubuntu-vm-testing.md
@@ -7,9 +7,9 @@ Supported Ubuntu LTSes:
| Release | Codename | SSH port | Live suite | Status |
|---|---|---|---|---|
-| 22.04 | jammy | 2222 | 79/79 | validated |
-| 24.04 | noble | 2223 | 79/79 | validated (default) |
-| 26.04 | resolute | 2224 | 79/79 | validated |
+| 22.04 | jammy | 2222 | 79/79 | validated |
+| 24.04 | noble | 2223 | 79/79 | validated (default) |
+| 26.04 | resolute | 2224 | 79/79 | validated |
Each release has a record run and a `.replay.json` twin in
`tests/evidence/story-runs/`, and `tests/release/cassette-replay-parity.test.sh`
diff --git a/docs/developer-guide.md b/docs/developer-guide.md
index 32905396..c8036ba9 100644
--- a/docs/developer-guide.md
+++ b/docs/developer-guide.md
@@ -282,6 +282,13 @@ and the workspace test baseline. `scripts/test_baseline.sh` runs
`cargo nextest run --workspace --locked` and checks the measured test count.
Run `bash .githooks/pre-commit` to invoke the same gate manually.
+Never type a figure into the README or the docs. Add your tests, run
+`UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh`, and commit the evidence file
+and whatever it re-rendered. The command records the count and then runs
+`scripts/render_figures.py --write`, which rewrites every `fig` marker from its
+source. The marker syntax is explained in
+[docs/contributing/testing.md](contributing/testing.md#published-figures).
+
The secret screen recognizes the credential formats of every key-bearing
provider in the setup wizard, including OpenAI service-account and `None`
keys, DeepSeek's hexadecimal keys, and xAI keys. Mistral's keys have no
@@ -367,7 +374,7 @@ crates/
sysknife-brain/ LLM planner, provider adapters, safety fence
sysknife-types/ Shared domain types (CallerRole, RiskLevel, JobState, …)
sysknife-core/ Config loading, shared constants
- sysknife-daemon/ Privileged executor, 191 actions with an `ActionSpec`,
+ sysknife-daemon/ Privileged executor, one `ActionSpec` per action,
IPC, rollback, SQLite
sysknife-proto/ Protobuf definitions (future use)
sysknife-daemon-test/ TAP integration harness for the daemon IPC layer (VM only)
diff --git a/docs/distro-support.md b/docs/distro-support.md
index e78e7ff3..c53e67ec 100644
--- a/docs/distro-support.md
+++ b/docs/distro-support.md
@@ -56,9 +56,9 @@ tracked separately in [#239](https://github.com/lacs-project/sysknife/issues/239
| Distro | Action backend | Evidence | Launch tier |
|---|---|---|---|
-| **Ubuntu 24.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-24.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok` | **Validated** |
-| **Ubuntu 22.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-22.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok`. This is the release whose twin exercises the recorded-rejection path: story 101's first call was refused by the provider (`tool_use_failed`), and the replay serves the refusal, the correction and the answer, 81 calls for 79 stories | **Validated** |
-| **Ubuntu 26.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-26.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok`; plus sudo-rs sudoers verification (26.04 ships sudo-rs 0.2.x; `visudo -cf` parses the SysKnife sudoers and every grant — including the trailing-`*` wildcard grants — is honoured) | **Validated** |
+| **Ubuntu 24.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-24.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok` | **Validated** |
+| **Ubuntu 22.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-22.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok`. This is the release whose twin exercises the recorded-rejection path: story 101's first call was refused by the provider (`tool_use_failed`), and the replay serves the refusal, the correction and the answer, 81 calls for 79 stories | **Validated** |
+| **Ubuntu 26.04 LTS** | apt, ufw, netplan, snap, AppArmor, systemd, containers | live-VM story suite, **79/79**, recorded in `ubuntu-26.04-gpt-oss-120b.json` and fully reproduced by its `.replay.json` twin — zero misses, `cassette_audit.verdict` `ok`; plus sudo-rs sudoers verification (26.04 ships sudo-rs 0.2.x; `visudo -cf` parses the SysKnife sudoers and every grant — including the trailing-`*` wildcard grants — is honoured) | **Validated** |
| **Every other Ubuntu 20.04+ release** (20.04, 20.10, 21.x, 23.x, 25.x, 26.10, …) | Ubuntu/apt family | Eligible by release family; no per-release VM run | **Smoke-tested** |
| **Fedora Silverblue 44** | rpm-ostree, Flatpak, toolbox, firewalld, systemd, containers | Harness and fixture coverage; no live-VM run on this release | **Blocked** (`make install` does not complete on rpm-ostree, [#301](https://github.com/lacs-project/sysknife/issues/301)) |
| **Other Fedora Atomic 41+ variants** | rpm-ostree family | Detection and shared action tests, plus a live VM run on Fedora 43 Silverblue (2026-08-24) that provisioned and then stopped at `make install` | **Blocked** (`make install` does not complete on rpm-ostree, [#301](https://github.com/lacs-project/sysknife/issues/301)) |
@@ -94,7 +94,7 @@ family and the atomic story family are implemented and covered by the workspace
suite. What is missing is a way to put the helpers somewhere the daemon's own
grants already point.
-The deterministic workspace baseline is 2,011 Rust tests plus 72 frontend
+The deterministic workspace baseline is 2,000+ Rust tests plus 70+ frontend
tests. Those tests verify action construction, policy, approval, storage, and
UI behavior, but they do not replace a real distribution VM run.
diff --git a/docs/introduction.md b/docs/introduction.md
index 66acd5ef..d181cf3b 100644
--- a/docs/introduction.md
+++ b/docs/introduction.md
@@ -106,10 +106,10 @@ SysKnife, not an afterthought.
> **ℹ️ Distro support**
>
-> The current suite is 99 Ubuntu stories. All three Ubuntu LTS releases have a
-> committed live-VM run covering 79 of those Ubuntu stories, in
+> The current suite is 90+ Ubuntu stories. All three Ubuntu LTS releases have a
+> committed live-VM run covering 79 of those Ubuntu stories, in
> `tests/evidence/story-runs/`:
-> 22.04, 24.04 and 26.04 all at 79/79. The twenty additional stories are not yet
+> 22.04, 24.04 and 26.04 all at 79/79. The 20+ additional stories are not yet
> included in a committed live-VM run. Each run has a replay twin that
> reproduces it, serving every call with zero misses.
> Every Debian-only action has a story.
@@ -146,7 +146,7 @@ flow.
## Status
-192 typed actions · 2,011 Rust tests + 72 frontend tests · MIT
+190+ typed actions · 2,000+ Rust tests + 70+ frontend tests · MIT
SysKnife is the reference implementation of the
[LACS specification](https://github.com/lacs-project/specification) — a
diff --git a/docs/quickstart.md b/docs/quickstart.md
index 5957d706..6ba53872 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -72,7 +72,7 @@ build (measured 6m56s on Ubuntu 24.04, 11m43s on 22.04).
> **ℹ️ Fedora / Silverblue**
>
> Ubuntu 22.04, 24.04 and 26.04 are each validated by a committed live-VM
-> story run (79/79) with a replay twin; other Ubuntu releases from 20.04 up are
+> story run (79/79) with a replay twin; other Ubuntu releases from 20.04 up are
> eligible without a per-release run. Fedora Atomic uses the rpm-ostree action
> family and requires a current Silverblue 44 validation run for each release.
> Plain Fedora remains experimental. See [distro support](distro-support.md).
diff --git a/docs/release-readiness.md b/docs/release-readiness.md
index 3d614979..de77ee24 100644
--- a/docs/release-readiness.md
+++ b/docs/release-readiness.md
@@ -12,7 +12,7 @@ Run these on the exact candidate commit and retain the CI links:
- [ ] `cargo fmt --all --check`
- [ ] `cargo clippy --workspace --all-features --all-targets --locked -- -D warnings`
- [ ] `cargo nextest run --workspace --locked`
-- [ ] frontend typecheck, 72 tests, production build, and production npm audit
+- [ ] frontend typecheck, 70+ tests, production build, and production npm audit
- [ ] live PostgreSQL migration and store contract
- [ ] ShellCheck, YAML lint, repository completeness, release versions, public
claims, and bootstrap contracts
diff --git a/docs/testing/user-stories.md b/docs/testing/user-stories.md
index e1d1df8f..7ffdc39f 100644
--- a/docs/testing/user-stories.md
+++ b/docs/testing/user-stories.md
@@ -5,8 +5,8 @@ Fedora Atomic Desktop (Silverblue, Kinoite, Sway Atomic, Budgie Atomic, or
COSMIC Atomic). Run inside a QEMU/KVM VM via `tests/e2e/atomic-vm.sh`, or on
real hardware.
-These twenty are a readable subset, not the suite: the atomic family is 54
-stories and the Ubuntu family a further 99 (79 of them in the recorded
+These twenty are a readable subset, not the suite: the atomic family is 50+
+stories and the Ubuntu family a further 90+ (79 of them in the recorded
live-VM runs), all in `tests/e2e/stories/`. The
harness runs the family; this file explains twenty of them in prose.
@@ -500,7 +500,7 @@ by the manual QA checklist (see `demo-script.md`):
## Running the stories
-### Locally (real Silverblue VM, all 54 atomic stories)
+### Locally (real Silverblue VM, all 50+ atomic stories)
```sh
# One-time: download the Fedora Silverblue ISO and install it in QEMU/KVM
diff --git a/docs/typed-actions.md b/docs/typed-actions.md
index a19cdd04..d4e8a5ac 100644
--- a/docs/typed-actions.md
+++ b/docs/typed-actions.md
@@ -62,7 +62,7 @@ codebase. On the daemon side (`sysknife-daemon`), each action is backed by an
| `reboot_required` | Whether the daemon should warn the caller before proceeding |
| `rollback_available` | Whether a failure triggers automatic rollback |
-As of this writing the catalogue defines **192 actions** across families such
+As of this writing the catalogue defines **190+ actions** across families such
as Deployment, Services, Package Layering, Flatpak, Containers, Toolbox,
Network, Identity, SSH Keys, Package Repositories, apt/snap/ufw/netplan/grub
and AppArmor/cloud-init (Debian-family), Ubuntu Pro (Ubuntu-only), and
diff --git a/scripts/check_evidence_claims.py b/scripts/check_evidence_claims.py
index b1bc6e46..2c3c145e 100755
--- a/scripts/check_evidence_claims.py
+++ b/scripts/check_evidence_claims.py
@@ -18,6 +18,20 @@
A story claim additionally has to name a *full* family run. Subset runs record an
empty `story_set`, so a four-story probe can never be quoted as the headline.
+Figures are not typed. Every figure these checks match in CLAIM_FILES must sit
+inside a marker that scripts/render_figures.py owns:
+
+ VALUE exact ("2,004")
+ VALUE rounded floor ("2,000+")
+
+This module refuses a matched figure outside a marker (check_unmarked_figures),
+so a hand-typed number cannot creep back, and it keeps the semantic checks
+(a story claim must name a full family run, validated tiers, ...) running on the
+rendered text, with the markers unwrapped. Freshness of the marked values is
+render_figures.py --check; `UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh`
+re-renders them. The marker grammar lives here, once, and render_figures.py
+imports it together with the value readers below.
+
Run via scripts/check_public_claims.sh; tests/release/public-claims.test.sh
mutates each claim and asserts this rejects it.
"""
@@ -119,6 +133,231 @@ def read_claim_files(root: Path) -> dict[str, str]:
)
return texts
+# ---------------------------------------------------------------------------
+# Figure markers. The grammar has one implementation, here; render_figures.py
+# imports it.
+#
+# VALUE exact
+# VALUE rounded floor, rendered "N+"
+#
+# HTML comments are invisible on GitHub and in mdBook. A marker sits on one line.
+# Markers inside a fenced code block are refused (a fence is shown literally, so
+# the comments would be visible and the value would never be read as a figure);
+# markers inside an inline code span are documentation of the syntax and are
+# ignored.
+# ---------------------------------------------------------------------------
+MARKER_START = re.compile(r"")
+MARKER_OPEN_REST = re.compile(
+ r"^:(?P[A-Za-z0-9_.:-]+)(?:\s+floor=(?P[0-9]+))?$"
+)
+CODE_FENCE = re.compile(r"^\s*(```|~~~)")
+INLINE_CODE = re.compile(r"`[^`\n]*`")
+
+
+class Marker:
+ """One well-formed marker: where it sits and what it currently says."""
+
+ __slots__ = ("name", "floor", "start", "end", "value_start", "value_end", "line", "value")
+
+ def __init__(self, name, floor, start, end, value_start, value_end, line, value):
+ self.name = name
+ self.floor = floor
+ self.start = start
+ self.end = end
+ self.value_start = value_start
+ self.value_end = value_end
+ self.line = line
+ self.value = value
+
+
+def find_markers(text: str) -> tuple[list[Marker], list[tuple[int, str]]]:
+ """Return the well-formed markers and every structural problem as (line, message).
+
+ Problems: a marker inside a code fence, an unparseable marker comment, an
+ unbalanced or nested pair, a pair split across lines, a floor of zero.
+ """
+ problems: list[tuple[int, str]] = []
+ tokens = [] # (is_close, name, floor, start, end, line)
+ in_fence = False
+ offset = 0
+ for lineno, line in enumerate(text.splitlines(keepends=True), 1):
+ line_start = offset
+ offset += len(line)
+ if CODE_FENCE.match(line):
+ in_fence = not in_fence
+ continue
+ if in_fence:
+ if MARKER_START.search(line):
+ problems.append((lineno, "figure marker inside a fenced code block"))
+ continue
+ spans = [m.span() for m in INLINE_CODE.finditer(line)]
+ for start in MARKER_START.finditer(line):
+ if any(a <= start.start() < b for a, b in spans):
+ continue
+ token = MARKER_TOKEN.match(line, start.start())
+ if token is None:
+ problems.append((lineno, "malformed figure marker (no closing '-->')"))
+ continue
+ rest = token.group("rest")
+ if token.group("close"):
+ if rest.strip():
+ problems.append((lineno, f"malformed closing marker {token.group(0)!r}"))
+ continue
+ tokens.append((True, None, None, line_start + token.start(),
+ line_start + token.end(), lineno))
+ continue
+ opened = MARKER_OPEN_REST.match(rest.strip())
+ if opened is None:
+ problems.append((lineno, f"malformed figure marker {token.group(0)!r}"))
+ continue
+ floor = opened.group("floor")
+ if floor is not None and int(floor) == 0:
+ problems.append((lineno, "floor=0 is not a rounding"))
+ continue
+ tokens.append((False, opened.group("name"),
+ int(floor) if floor is not None else None,
+ line_start + token.start(), line_start + token.end(), lineno))
+
+ markers: list[Marker] = []
+ pending = None
+ for token in tokens:
+ is_close, name, floor, start, end, lineno = token
+ if not is_close:
+ if pending is not None:
+ problems.append((lineno, f"nested figure marker inside the one opened on line {pending[5]}"))
+ pending = token
+ continue
+ pending = token
+ continue
+ if pending is None:
+ problems.append((lineno, "closing figure marker without an opening one"))
+ continue
+ _, name, floor, open_start, open_end, open_line = pending
+ pending = None
+ value = text[open_end:start]
+ if "\n" in value:
+ problems.append((open_line, "figure marker spans lines"))
+ continue
+ markers.append(Marker(name, floor, open_start, end, open_end, start, open_line, value))
+ if pending is not None:
+ problems.append((pending[5], "unbalanced figure marker: opened and never closed"))
+ return markers, problems
+
+
+def unwrap_markers(text: str) -> str:
+ """The text a reader sees: each marker replaced by its value."""
+ markers, _ = find_markers(text)
+ out, last = [], 0
+ for marker in markers:
+ out.append(text[last : marker.start])
+ out.append(marker.value)
+ last = marker.end
+ out.append(text[last:])
+ return "".join(out)
+
+
+def mask_markers(text: str) -> str:
+ """The text with every marker replaced by a placeholder, so what is left is
+ exactly the figures nothing owns."""
+ markers, _ = find_markers(text)
+ out, last = [], 0
+ for marker in markers:
+ out.append(text[last : marker.start])
+ out.append("FIG")
+ last = marker.end
+ out.append(text[last:])
+ return "".join(out)
+
+
+# The README badge URLs are not figures and need no exemption: they are
+# percent-encoded (`label=Rust%20tests`), so no shape below can match inside one.
+# The pristine tree passing with both badges present is the proof.
+# Digits followed by a floor's trailing plus ("2,000+ Rust tests"): read as the
+# bare number so the shapes below match a hand-typed floor exactly as they match
+# a hand-typed exact figure.
+FLOOR_PLUS = re.compile(r"(?<=[0-9])\+(?=[\s-])")
+
+# The shapes the checks below match. check_figure builds its pattern from these
+# nouns; the rest are the patterns of the check named beside each one.
+FIGURE_NOUNS = ("Rust tests", "frontend tests", "typed actions")
+BARE_TOTAL_RE = re.compile(r"([0-9][0-9,]*)\s+tests\b") # check_bare_test_totals
+STORY_RATE_RE = re.compile(r"([0-9]+)\s*/\s*([0-9]+)\s+stories") # check_story_claims
+BARE_STORY_RE = re.compile(r"(?[0-9]{2,})\s+(?Patomic|ubuntu)\b", re.IGNORECASE
+) # check_bare_story_counts
+ACTION_BARE_RE = re.compile(r"\b([0-9]{2,})\s+(?:typed\s+)?actions\b", re.IGNORECASE)
+ACTION_SUBSET_RE = re.compile(
+ r"\b([0-9]+)\s+actions\s+(?:with|have)\s+an\s+`?ActionSpec`?\b", re.IGNORECASE
+) # check_action_figures
+# check_story_coverage_claims reads these three counts out of one sentence; each
+# is a figure on its own.
+UNCOVERED_FIGURE_RES = (
+ re.compile(r"\b[0-9]+\s+are\s+still\s+untouched\s+by\s+any\s+story\b", re.IGNORECASE),
+ re.compile(r"\b[0-9]+\s+Fedora-only\s+and\b", re.IGNORECASE),
+ re.compile(r"\b[0-9]+\s+Ubuntu-only\s+ones\b", re.IGNORECASE),
+)
+
+
+def _figure_shapes(text: str) -> list[tuple[int, int]]:
+ """Every (start, end) span any figure check would match in `text`."""
+ hits: list[tuple[int, int]] = []
+ for noun in FIGURE_NOUNS:
+ spaced = r"\s+".join(re.escape(word) for word in noun.split())
+ for match in re.finditer(rf"([0-9][0-9,]*)\s+{spaced}", text):
+ hits.append(match.span())
+ for match in BARE_TOTAL_RE.finditer(text):
+ if int(match.group(1).replace(",", "")) >= BARE_TOTAL_FLOOR:
+ hits.append(match.span())
+ for regex in (STORY_RATE_RE, BARE_STORY_RE, FAMILY_RE, ACTION_BARE_RE, ACTION_SUBSET_RE,
+ *UNCOVERED_FIGURE_RES):
+ for match in regex.finditer(text):
+ hits.append(match.span())
+ return hits
+
+
+def check_unmarked_figures(raw_texts: dict[str, str]) -> list[str]:
+ """Refuse a figure that the checks above would match but no marker owns.
+
+ The marked figures belong to render_figures.py --check. Without this rule a
+ hand-typed number would still be validated by the semantic checks, but it
+ would be one more number a contributor has to edit by hand, and a floor typed
+ by hand would not be validated at all (a floor is skipped by those checks, as
+ `2,000+` is not an exact count).
+ """
+ problems = []
+ for rel, text in raw_texts.items():
+ _, structural = find_markers(text)
+ for lineno, message in structural:
+ problems.append(f"{rel}:{lineno}: {message}")
+ masked = mask_markers(text)
+ scanned = FLOOR_PLUS.sub("", masked)
+ # Offsets in `scanned` map back into `masked` by adding the floor-plus
+ # characters removed before them, so a hand-typed "2,000+" is quoted as typed.
+ removed = [m.start() for m in FLOOR_PLUS.finditer(masked)]
+
+ def original(offset: int) -> int:
+ for index, position in enumerate(removed):
+ if position - index >= offset:
+ return offset + index
+ return offset + len(removed)
+
+ seen = set()
+ for start, end in sorted(_figure_shapes(scanned)):
+ lineno = scanned.count("\n", 0, start) + 1
+ matched = masked[original(start) : original(end)]
+ key = (lineno, " ".join(matched.split()))
+ if key in seen:
+ continue
+ seen.add(key)
+ problems.append(
+ f"{rel}:{lineno}: figure {' '.join(matched.split())!r} is not inside a "
+ " marker. Do not type figures: wrap it and run "
+ "scripts/render_figures.py --write"
+ )
+ return problems
+
def count_actions(root: Path) -> int:
"""Count entries in the brain's KNOWN_ACTIONS table."""
@@ -167,9 +406,8 @@ def check_figure(texts: dict[str, str], noun: str, expected: int) -> list[str]:
def check_bare_test_totals(texts: dict[str, str], allowed: tuple[int, ...]) -> list[str]:
"""Flag an unqualified ` tests` that looks like a stale workspace total."""
problems = []
- pattern = re.compile(r"([0-9][0-9,]*)\s+tests\b")
for rel, text in texts.items():
- for match in pattern.finditer(text):
+ for match in BARE_TOTAL_RE.finditer(text):
# Skip the qualified forms; check_figure owns those.
preceding = text[max(0, match.start() - 12) : match.start()]
claimed = int(match.group(1).replace(",", ""))
@@ -211,10 +449,9 @@ def story_run_has_no_unproven_results(run: dict) -> bool:
def check_story_claims(texts: dict[str, str], runs: list[dict]) -> list[str]:
problems = []
- pattern = re.compile(r"([0-9]+)\s*/\s*([0-9]+)\s+stories")
for rel, text in texts.items():
for line in text.splitlines():
- match = pattern.search(line)
+ match = STORY_RATE_RE.search(line)
if not match:
continue
passed, total = int(match.group(1)), int(match.group(2))
@@ -470,8 +707,8 @@ def check_story_coverage_claims(texts: dict[str, str], root: Path) -> list[str]:
problems = []
family_claim = re.search(
- r"\bThe suite is\s+(?P\d+)\s+stories?\s*:\s*"
- r"(?P\d+)\s+atomic\s*\+\s*(?P\d+)\s+Ubuntu\b",
+ r"\bThe suite is\s+(?P\d+\+?)\s+stories?\s*:\s*"
+ r"(?P\d+\+?)\s+atomic\s*\+\s*(?P\d+\+?)\s+Ubuntu\b",
text,
re.IGNORECASE,
)
@@ -481,17 +718,19 @@ def check_story_coverage_claims(texts: dict[str, str], root: Path) -> list[str]:
"'The suite is N stories: A atomic + U Ubuntu'"
)
else:
+ # A rounded floor ("150+") is owned by render_figures.py --check; only an
+ # exact count is compared here.
published_sizes = {
- "total": int(family_claim.group("total")),
- "atomic": int(family_claim.group("atomic")),
- "ubuntu": int(family_claim.group("ubuntu")),
+ label: int(family_claim.group(label))
+ for label in ("total", "atomic", "ubuntu")
+ if not family_claim.group(label).endswith("+")
}
derived_sizes = {
"total": total,
"atomic": family_sizes.get("atomic", 0),
"ubuntu": family_sizes.get("ubuntu", 0),
}
- for label in ("total", "atomic", "ubuntu"):
+ for label in published_sizes:
if published_sizes[label] != derived_sizes[label]:
problems.append(
f"{rel}: story coverage {label} count has published "
@@ -611,10 +850,8 @@ def check_bare_story_counts(
individually legitimate sizes is not enough, since swapped labels would
otherwise pass.
"""
- bare = re.compile(r"(?[0-9]{2,})\s+(?Patomic|ubuntu)\b", re.IGNORECASE
- )
+ bare = BARE_STORY_RE
+ family = FAMILY_RE
allowed = {
run.get("totals", {}).get("total")
@@ -684,11 +921,8 @@ def check_action_figures(
writable, forces it to carry its qualifier, and still catches the count
drifting.
"""
- bare = re.compile(r"\b([0-9]{2,})\s+(?:typed\s+)?actions\b", re.IGNORECASE)
- subset = re.compile(
- r"\b([0-9]+)\s+actions\s+(?:with|have)\s+an\s+`?ActionSpec`?\b",
- re.IGNORECASE,
- )
+ bare = ACTION_BARE_RE
+ subset = ACTION_SUBSET_RE
problems = []
for rel, text in texts.items():
@@ -814,6 +1048,90 @@ def load_test_baseline(root: Path) -> dict:
return baseline
+LTS_RELEASES = ("22.04", "24.04", "26.04")
+
+
+def story_run_totals(root: Path, stem: str) -> dict:
+ """The totals of one recorded run, addressed by its file name stem."""
+ path = root / STORY_RUNS / f"{stem}.json"
+ if not path.is_file():
+ raise Failure(f"no recorded story run {STORY_RUNS}/{stem}.json")
+ try:
+ run = json.loads(path.read_text(encoding="utf-8"))
+ except (OSError, json.JSONDecodeError) as exc:
+ raise Failure(f"{STORY_RUNS}/{stem}.json is not readable JSON: {exc}") from exc
+ totals = run.get("totals") if isinstance(run, dict) else None
+ if not isinstance(totals, dict) or not all(
+ isinstance(totals.get(f), int) and not isinstance(totals.get(f), bool)
+ for f in ("passed", "total")
+ ):
+ raise Failure(f"{STORY_RUNS}/{stem}.json has no integer totals.passed and totals.total")
+ return totals
+
+
+def story_run_rate(root: Path, stem: str) -> str:
+ """`passed/total` of one recorded run, read from its artifact."""
+ totals = story_run_totals(root, stem)
+ return f"{totals['passed']}/{totals['total']}"
+
+
+def story_run_total(root: Path, stem: str) -> int:
+ """Stories in one recorded run, read from its artifact."""
+ return story_run_totals(root, stem)["total"]
+
+
+def story_run_served(root: Path, stem: str) -> int:
+ """Calls a replay served, from its `cassette_audit`."""
+ path = root / STORY_RUNS / f"{stem}.json"
+ story_run_totals(root, stem) # missing or unreadable: fail with the reader's message
+ audit = json.loads(path.read_text(encoding="utf-8")).get("cassette_audit")
+ served = audit.get("served") if isinstance(audit, dict) else None
+ if not isinstance(served, int) or isinstance(served, bool):
+ raise Failure(f"{STORY_RUNS}/{stem}.json has no integer cassette_audit.served")
+ return served
+
+
+def lts_story_rates(root: Path) -> dict[str, tuple[int, int]]:
+ """`(passed, total)` of the latest full-family record run of each Ubuntu LTS."""
+ latest: dict[str, tuple[str, tuple[int, int]]] = {}
+ for run in load_story_runs(root):
+ release = str(run.get("release", ""))
+ if (
+ release in LTS_RELEASES
+ and run.get("story_set") in FULL_STORY_SETS
+ and run.get("cassette_mode") == "record"
+ and story_run_has_no_unproven_results(run)
+ ):
+ totals = run.get("totals", {})
+ key = str(run.get("ran_at", ""))
+ if release not in latest or key > latest[release][0]:
+ latest[release] = (key, (totals.get("passed"), totals.get("total")))
+ missing = [release for release in LTS_RELEASES if release not in latest]
+ if missing:
+ raise Failure(
+ "no full-family record run is committed for Ubuntu " + ", ".join(missing)
+ )
+ return {release: latest[release][1] for release in LTS_RELEASES}
+
+
+def lts_story_total(root: Path) -> int:
+ """The story count every LTS run covers. Refuses when the runs disagree."""
+ rates = lts_story_rates(root)
+ totals = {total for _, total in rates.values()}
+ if len(totals) != 1:
+ raise Failure(f"the LTS runs cover different story counts: {rates}")
+ return totals.pop()
+
+
+def lts_story_rate(root: Path) -> str:
+ """`passed/total` shared by every LTS run. Refuses when the runs disagree."""
+ rates = lts_story_rates(root)
+ if len(set(rates.values())) != 1:
+ raise Failure(f"the LTS runs do not share one pass rate: {rates}")
+ passed, total = next(iter(rates.values()))
+ return f"{passed}/{total}"
+
+
def check_pre_commit_commands(root: Path, guide: str) -> list[str]:
"""Compare the documented gate with the executable hook, in order."""
hook = root / ".githooks/pre-commit"
@@ -869,11 +1187,14 @@ def check_framework_commands(root: Path) -> list[str]:
def main() -> int:
root = Path(sys.argv[1] if len(sys.argv) > 1 else ".").resolve()
try:
- texts = read_claim_files(root)
+ raw_texts = read_claim_files(root)
+ # The semantic checks read what a visitor reads: markers unwrapped.
+ texts = {rel: unwrap_markers(text) for rel, text in raw_texts.items()}
baseline = load_test_baseline(root)
problems = []
+ problems += check_unmarked_figures(raw_texts)
problems += check_pre_commit_commands(root, texts["docs/developer-guide.md"])
problems += check_framework_commands(root)
problems += check_figure(texts, "Rust tests", baseline["tests"])
@@ -894,12 +1215,12 @@ def main() -> int:
problems += check_validated_tiers(texts, root)
problems += check_action_figures(texts, count_actions(root), count_action_specs(root))
- expected_tests = f"{baseline['tests']:,} Rust tests"
for rel in REQUIRE_TEST_COUNT:
- if rel in texts and expected_tests not in texts[rel]:
+ names = [m.name for m in find_markers(raw_texts[rel])[0]]
+ if "rust_tests" not in names:
problems.append(
- f"{rel}: does not state the measured baseline "
- f"({expected_tests})"
+ f"{rel}: does not state the measured baseline; it needs a "
+ " marker"
)
if problems:
diff --git a/scripts/check_public_claims.sh b/scripts/check_public_claims.sh
index 615ee010..4848b93a 100755
--- a/scripts/check_public_claims.sh
+++ b/scripts/check_public_claims.sh
@@ -61,6 +61,14 @@ if ! python3 "$script_dir/check_evidence_claims.py" "$repo_root"; then
printf 'Invalid public claim: a published figure does not derive from evidence\n' >&2
exit 1
fi
+# Every figure marker must render from its source. check_evidence_claims.py has
+# already refused a figure outside a marker; this refuses a marker whose value
+# is stale, unknown, malformed or inside a code fence. Run on the same tree, with
+# no pipe, so its exit status is the one tested here.
+if ! python3 "$script_dir/render_figures.py" --check "$repo_root"; then
+ printf 'Invalid public claim: a figure marker does not render from its source\n' >&2
+ exit 1
+fi
reject_pattern 'until npm publish lands|publish[- ]pending' \
'setup package is documented as unpublished' "${claim_files[@]}"
# SECURITY.md once said every enforcement layer is independent and a bypass of
diff --git a/scripts/render_figures.py b/scripts/render_figures.py
new file mode 100644
index 00000000..1afa6a93
--- /dev/null
+++ b/scripts/render_figures.py
@@ -0,0 +1,188 @@
+#!/usr/bin/env python3
+"""Render the figures in SysKnife's README and docs from their sources.
+
+A figure that describes the project is never typed. It sits in a marker:
+
+ VALUE exact, thousands separators: 2,004
+ VALUE rounded floor: 2,000+
+
+and this script writes VALUE. HTML comments are invisible on GitHub and in mdBook.
+`floor=K` renders floor(v / K) * K with a trailing "+", and refuses when v < K
+(a floor of zero says nothing). A figure a routine PR moves is a floor; a figure
+only a deliberate act changes (a recorded story run) stays exact.
+
+ render_figures.py [--check] [ROOT] exit 1 naming file:line, the stored value and
+ the rendered one, for a stale marker, an
+ unknown NAME, a malformed marker (unbalanced,
+ nested) and a marker inside a code fence
+ render_figures.py --write [ROOT] rewrite only the marker contents; a second
+ run changes no byte
+
+Every value comes from a reader imported out of check_evidence_claims.py, the one
+place that knows how to read each source: one source per figure, never a second
+implementation here. A source that cannot be read, or a run that finds no marker
+at all in CLAIM_FILES, is an error and exits 2: nothing is skipped.
+
+`UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh` records the test evidence and then
+runs --write, so one command updates evidence and docs.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+from typing import Callable
+
+SCRIPT_DIR = Path(__file__).resolve().parent
+if str(SCRIPT_DIR) not in sys.path:
+ sys.path.insert(0, str(SCRIPT_DIR))
+
+import check_evidence_claims as sources # noqa: E402
+from check_evidence_claims import Failure, find_markers # noqa: E402
+
+# NAME -> reader. A NAME with an argument is `NAME:ARG` and its reader takes it.
+SIMPLE: dict[str, Callable[[Path], "int | str"]] = {
+ "rust_tests": lambda root: sources.load_test_baseline(root)["tests"],
+ "frontend_tests": lambda root: sources.load_test_baseline(root)["frontend_tests"],
+ "typed_actions": sources.count_actions,
+ "action_specs": sources.count_action_specs,
+ "stories_total": lambda root: sum(sources.story_family_sizes(root).values()),
+ "stories_atomic": lambda root: sources.story_family_sizes(root)["atomic"],
+ "stories_ubuntu": lambda root: sources.story_family_sizes(root)["ubuntu"],
+ "uncovered_all": lambda root: sources.uncovered_action_counts(root)["All"],
+ "uncovered_fedora": lambda root: sources.uncovered_action_counts(root)["Fedora"],
+ "uncovered_ubuntu": lambda root: sources.uncovered_action_counts(root)["Ubuntu"],
+ "lts_story_total": sources.lts_story_total,
+ "lts_story_rate": sources.lts_story_rate,
+ "stories_ubuntu_unrecorded": lambda root: (
+ sources.story_family_sizes(root)["ubuntu"] - sources.lts_story_total(root)
+ ),
+}
+PARAMETRIC: dict[str, Callable[[Path, str], "int | str"]] = {
+ "story_run": sources.story_run_rate,
+ "story_run_total": sources.story_run_total,
+ "story_run_served": sources.story_run_served,
+}
+
+
+def known_names() -> list[str]:
+ return sorted(SIMPLE) + [f"{name}:ARG" for name in sorted(PARAMETRIC)]
+
+
+def read_value(root: Path, name: str, cache: dict) -> "int | str":
+ """Read one figure's source. Raises KeyError for an unknown NAME and Failure
+ when the source cannot be read."""
+ if name in cache:
+ return cache[name]
+ if name in SIMPLE:
+ value = SIMPLE[name](root)
+ else:
+ base, _, arg = name.partition(":")
+ if base not in PARAMETRIC or not arg:
+ raise KeyError(name)
+ value = PARAMETRIC[base](root, arg)
+ cache[name] = value
+ return value
+
+
+def render(value: "int | str", floor: "int | None") -> str:
+ """Format a source value. Raises ValueError when a floor cannot be applied."""
+ if isinstance(value, bool) or not isinstance(value, (int, str)):
+ raise ValueError(f"source returned {value!r}")
+ if floor is None:
+ return f"{value:,}" if isinstance(value, int) else value
+ if not isinstance(value, int):
+ raise ValueError(f"floor={floor} needs a count, the source gave {value!r}")
+ if value < floor:
+ raise ValueError(f"value {value:,} is below floor={floor:,}; state it exactly or lower K")
+ return f"{value // floor * floor:,}+"
+
+
+def process(root: Path, write: bool) -> tuple[list[str], list[str], int, int]:
+ """Check (or rewrite) every marker in CLAIM_FILES.
+
+ Returns (problems, stale, marker count, files rewritten). A problem is what
+ --write cannot fix (unknown NAME, malformed marker, a floor that cannot apply,
+ no markers at all); stale markers are what it fixes. Nothing is written while
+ there is a problem.
+ """
+ texts = sources.read_claim_files(root)
+ cache: dict = {}
+ problems: list[str] = []
+ stale: list[str] = []
+ rewrites: dict[str, str] = {}
+ total = 0
+ for rel, text in texts.items():
+ markers, structural = find_markers(text)
+ problems += [f"{rel}:{line}: {message}" for line, message in structural]
+ total += len(markers)
+ pieces, last = [], 0
+ for marker in markers:
+ try:
+ value = read_value(root, marker.name, cache)
+ except KeyError:
+ problems.append(
+ f"{rel}:{marker.line}: unknown figure NAME {marker.name!r}; "
+ f"known: {', '.join(known_names())}"
+ )
+ continue
+ try:
+ rendered = render(value, marker.floor)
+ except ValueError as exc:
+ problems.append(f"{rel}:{marker.line}: figure {marker.name!r}: {exc}")
+ continue
+ if rendered != marker.value:
+ stale.append(
+ f"{rel}:{marker.line}: stale figure {marker.name!r}: stored "
+ f"{marker.value!r}, rendered {rendered!r}"
+ )
+ pieces.append(text[last : marker.value_start])
+ pieces.append(rendered)
+ last = marker.value_end
+ pieces.append(text[last:])
+ new_text = "".join(pieces)
+ if new_text != text:
+ rewrites[rel] = new_text
+ if total == 0:
+ problems.append("no figure markers found in CLAIM_FILES; refusing to treat that as success")
+ written = 0
+ if write and not problems:
+ for rel, new_text in rewrites.items():
+ with open(root / rel, "w", encoding="utf-8", newline="") as handle:
+ handle.write(new_text)
+ written += 1
+ return problems, stale, total, written
+
+
+def main(argv: list[str]) -> int:
+ args = [a for a in argv if not a.startswith("--")]
+ flags = {a for a in argv if a.startswith("--")}
+ if flags - {"--check", "--write"} or {"--check", "--write"} <= flags or len(args) > 1:
+ print("usage: render_figures.py [--check | --write] [ROOT]", file=sys.stderr)
+ return 2
+ write = "--write" in flags
+ root = Path(args[0] if args else SCRIPT_DIR.parent).resolve()
+ try:
+ problems, stale, total, written = process(root, write)
+ except Failure as exc:
+ print(f"Figures could not be rendered: {exc}", file=sys.stderr)
+ return 2
+ except (OSError, UnicodeDecodeError) as exc:
+ print(f"Figures could not be rendered: could not read or write a file: {exc}", file=sys.stderr)
+ return 2
+ if problems or (stale and not write):
+ print("Figure markers that do not render from their sources:", file=sys.stderr)
+ for problem in problems + stale:
+ print(f" - {problem}", file=sys.stderr)
+ if write:
+ print("Nothing was written.", file=sys.stderr)
+ return 1
+ if write:
+ print(f"Rendered {total} figure markers; rewrote {written} files.")
+ else:
+ print(f"All {total} figure markers match their sources.")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main(sys.argv[1:]))
diff --git a/scripts/test_baseline.sh b/scripts/test_baseline.sh
index 10924f72..cb89ea1a 100755
--- a/scripts/test_baseline.sh
+++ b/scripts/test_baseline.sh
@@ -15,6 +15,10 @@
# UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh # Rust workspace
# UPDATE_TEST_BASELINE=1 scripts/test_baseline.sh --frontend # vitest
#
+# Either one then runs scripts/render_figures.py --write, which re-renders every
+# figure marker in the README and docs from the evidence. Commit the evidence
+# file and whatever it re-rendered.
+#
# Without UPDATE_TEST_BASELINE the run verifies instead: a suite whose size no
# longer matches the recorded baseline fails, because a published claim derives
# from that baseline. Remaining arguments are forwarded to the runner, so this is
@@ -104,6 +108,10 @@ if [[ "${UPDATE_TEST_BASELINE:-0}" == "1" ]]; then
--count "$count"
printf '\ntest_baseline: recorded %s %s tests in %s\n' \
"$count" "$suite" "${artifact#"$repo_root"/}"
+ # The published figures derive from the evidence just recorded, so re-render
+ # them in the same step: one command updates evidence and docs. No pipe, so a
+ # failed render is this script's exit status.
+ python3 "$repo_root/scripts/render_figures.py" --write "$repo_root"
exit 0
fi
diff --git a/tests/release/public-claims.test.sh b/tests/release/public-claims.test.sh
index fa528a05..f2f452c1 100755
--- a/tests/release/public-claims.test.sh
+++ b/tests/release/public-claims.test.sh
@@ -173,6 +173,55 @@ if ! "$checker" "$fixture" >/dev/null 2>&1; then
exit 1
fi
+
+# What render_figures.py renders for one figure out of the REAL tree, so the
+# assertions below never retype a figure or its rounding.
+rendered() {
+ python3 - "$repo_root" "$@" <<'PYEOF'
+import sys
+from pathlib import Path
+sys.path.insert(0, sys.argv[1] + "/scripts")
+import render_figures
+root, name = Path(sys.argv[1]), sys.argv[2]
+floor = int(sys.argv[3]) if len(sys.argv) > 3 else None
+print(render_figures.render(render_figures.read_value(root, name, {}), floor))
+PYEOF
+}
+
+# Replace the VALUE of the first marker NAME in a file and print the old value.
+# Exits non-zero when no such marker exists, so a mutation cannot silently not
+# apply.
+set_marker_value() {
+ python3 - "$1" "$2" "$3" <<'PYEOF'
+import re
+import sys
+from pathlib import Path
+path, name, new = Path(sys.argv[1]), sys.argv[2], sys.argv[3]
+text = path.read_text(encoding="utf-8")
+pattern = re.compile(
+ r"()([^<\n]*)()"
+)
+match = pattern.search(text)
+if not match:
+ raise SystemExit(f"no marker {name} in {path}")
+path.write_text(text[:match.start(2)] + new + text[match.end(2):], encoding="utf-8")
+print(match.group(2))
+PYEOF
+}
+
+# Put one named claim file back from the real tree and prove the copy is exact:
+# a restore that is not byte-identical would contaminate the next mutation.
+restore_claim_file() {
+ local rel="$1" want got
+ cp "$repo_root/$rel" "$fixture/$rel"
+ want="$(sha256sum < "$repo_root/$rel")"
+ got="$(sha256sum < "$fixture/$rel")"
+ if [[ "$want" != "$got" ]]; then
+ printf 'FAIL: restore of %s is not byte-identical\n' "$rel" >&2
+ exit 1
+ fi
+}
+
assert_rejected_with_diagnostic() {
local label="$1"
shift
@@ -247,9 +296,10 @@ from pathlib import Path
path = Path(sys.argv[1])
text = path.read_text(encoding="utf-8")
+# The figure sits inside a marker; mutate the number, not the comment.
pattern = re.compile(
- r"(of the action names available on both families,\s*)(\d+)"
- r"(\s+are still untouched by any story)"
+ r"(of the action names available on both families,\s*(?:)?)(\d+)"
+ r"((?:)?\s+are still untouched by any story)"
)
match = pattern.search(text)
if not match:
@@ -266,7 +316,7 @@ if [[ -z "${derived_all:-}" || -z "${published_all:-}" ]]; then
printf 'FAIL: story-coverage mutation produced no values\n' >&2
exit 1
fi
-if ! grep -Fq "$published_all are still untouched by any story" \
+if ! grep -Fq "${published_all} are still untouched by any story" \
"$fixture/CONTRIBUTING.md"; then
printf 'FAIL: story-coverage mutation did not apply\n' >&2
exit 1
@@ -707,31 +757,33 @@ assert_rejected() {
fi
}
-# Derive the baseline from the evidence artifact rather than repeating the
-# literal here. When the baseline moved, this mutation silently became a no-op:
-# the sed matched nothing, the checker legitimately passed, and the assertion
-# below then blamed the checker. A test whose mutation can stop mutating is worse
-# than no test — so every mutation below is confirmed to have applied.
+# A typed test count is refused twice over: as a figure outside a marker, and,
+# for an exact figure, because it disagrees with the evidence artifact. Derive the
+# wrong value from the artifact rather than repeating a literal; a test whose
+# mutation can stop mutating is worse than no test, so every mutation below is
+# confirmed to have applied.
baseline="$(python3 -c "
import json,sys
-print(f\"{json.load(open(sys.argv[1]))['tests']:,}\")
+print(f\"{json.load(open(sys.argv[1]))['tests'] + 1:,}\")
" "$repo_root/tests/evidence/workspace-tests.json")"
if [[ -z "$baseline" ]]; then
printf 'FAIL: could not read the test count from the evidence artifact\n' >&2
exit 1
fi
-sed -i "s/${baseline} Rust tests/1,256 Rust tests/" "$fixture/README.md"
-if ! grep -q '1,256 Rust tests' "$fixture/README.md"; then
+printf '\n%s Rust tests are typed here by hand.\n' "$baseline" >> "$fixture/README.md"
+if ! grep -Fq "$baseline Rust tests are typed here" "$fixture/README.md"; then
printf 'FAIL: baseline mutation did not apply; fixture would pass vacuously\n' >&2
exit 1
fi
-assert_rejected 'test count that disagrees with the evidence artifact'
+assert_rejected_with_diagnostic 'test count that disagrees with the evidence artifact' \
+ "README.md:" "claims $baseline Rust tests, evidence says" "is not inside a"
cp "$repo_root/README.md" "$fixture/README.md"
# A figure with no artifact at all behind it. This is the "65/65 stories" shape:
# published for months, never produced by a run.
printf '\nUbuntu 24.04 is validated with 47/50 stories on a live VM.\n' >> "$fixture/README.md"
-assert_rejected 'story pass rate with no recorded run'
+assert_rejected_with_diagnostic 'story pass rate with no recorded run' \
+ 'claims 47/50 stories with no run' 'to back it'
cp "$repo_root/README.md" "$fixture/README.md"
# ...and the same claim IS accepted once a run records it. Without this the rule
@@ -757,13 +809,20 @@ write_fixture_run() {
EV_SKIPPED=0 EV_RATELIMITED=0 \
python3 "$repo_root/scripts/record_story_run.py"
}
+# A story pass rate is claimed inside a marker, so the claim reaches the semantic
+# rules below with nothing else to reject it: the marker reads the run's own
+# totals, and only "does a full-family run back this" stands between the claim and
+# the checker. `fixture_claim` writes the claim line for the fixture run.
+fixture_claim() {
+ printf '\nUbuntu 24.04 is validated with %s stories on a live VM.\n' "$1"
+}
write_fixture_run ubuntu 50 47 ok
# The writer's output must be valid JSON — the defect it replaced was not.
python3 -m json.tool "$fixture/tests/evidence/story-runs/fixture-run.json" >/dev/null \
|| { printf 'FAIL: record_story_run.py emitted invalid JSON\n' >&2; exit 1; }
-printf '\nUbuntu 24.04 is validated with 47/50 stories on a live VM.\n' >> "$fixture/README.md"
-if ! "$checker" "$fixture" >/dev/null 2>&1; then
- printf 'FAIL: checker rejected a story claim that its recorded run backs\n' >&2
+fixture_claim 47/50 >> "$fixture/README.md"
+if ! output="$("$checker" "$fixture" 2>&1)"; then
+ printf 'FAIL: checker rejected a story claim that its recorded run backs\n%s\n' "$output" >&2
exit 1
fi
@@ -783,7 +842,8 @@ grep -q '"skipped": 1' "$fixture/tests/evidence/story-runs/fixture-run.json" ||
printf 'FAIL: skipped-story mutation did not apply\n' >&2
exit 1
}
-assert_rejected 'story pass rate backed by skipped stories'
+assert_rejected_with_diagnostic 'story pass rate backed by skipped stories' \
+ 'claims 47/50 stories with no run' 'to back it'
write_fixture_run ubuntu 50 47 ok
# Rate-limited stories are a separate diagnostic, but they are just as unable
@@ -801,30 +861,34 @@ grep -q '"rate_limited": 1' "$fixture/tests/evidence/story-runs/fixture-run.json
printf 'FAIL: rate-limited-story mutation did not apply\n' >&2
exit 1
}
-assert_rejected 'story pass rate backed by rate-limited stories'
+assert_rejected_with_diagnostic 'story pass rate backed by rate-limited stories' \
+ 'claims 47/50 stories with no run' 'to back it'
write_fixture_run ubuntu 50 47 ok
# Backed by a run, but attributed to a model that did not produce it.
-sed -i 's|47/50 stories on a live VM.|47/50 stories on a live VM with gpt-4.1.|' "$fixture/README.md"
+sed -i 's|47/50 stories on a live VM.|47/50 stories on a live VM with gpt-4.1.|' "$fixture/README.md"
grep -q 'gpt-4.1' "$fixture/README.md" || {
printf 'FAIL: model-attribution mutation did not apply\n' >&2
exit 1
}
-assert_rejected 'story pass rate attributed to the wrong model'
+assert_rejected_with_diagnostic 'story pass rate attributed to the wrong model' \
+ 'attributes 47/50 stories to gpt-4.1'
cp "$repo_root/README.md" "$fixture/README.md"
# A replay the harness declared unproven must not back a figure either. The
# artifact used to be written before the cassette audit ran, so a run that missed
# every call still produced a file with a healthy pass rate.
write_fixture_run ubuntu 50 47 failed
-printf '\nUbuntu 24.04 is validated with 47/50 stories on a live VM.\n' >> "$fixture/README.md"
-assert_rejected 'story pass rate from a replay whose cassette audit failed'
+fixture_claim 47/50 >> "$fixture/README.md"
+assert_rejected_with_diagnostic 'story pass rate from a replay whose cassette audit failed' \
+ 'claims 47/50 stories with no run' 'to back it'
cp "$repo_root/README.md" "$fixture/README.md"
# A subset run must never back a headline: probes record an empty story_set.
write_fixture_run "" 4 4 ok
-printf '\nUbuntu 24.04 is validated with 4/4 stories on a live VM.\n' >> "$fixture/README.md"
-assert_rejected 'four-story probe quoted as a headline figure'
+fixture_claim 4/4 >> "$fixture/README.md"
+assert_rejected_with_diagnostic 'four-story probe quoted as a headline figure' \
+ 'claims 4/4 stories with no run' 'to back it'
cp "$repo_root/README.md" "$fixture/README.md"
rm -f "$fixture/tests/evidence/story-runs/fixture-run.json"
@@ -834,32 +898,21 @@ assert_rejected 'a claim file missing from the checked surface'
mv "$fixture/ROADMAP.held" "$fixture/ROADMAP.md"
# The action count is derived from the catalogue source, so a retyped one fails.
-actions="$(grep -oE '[0-9]+ typed actions' "$fixture/docs/introduction.md" | head -1 | grep -oE '^[0-9]+' || true)"
-if [[ -z "$actions" ]]; then
- printf 'FAIL: could not find the typed-action count in the fixture\n' >&2
+printf '\nThe catalogue holds 999 typed actions.\n' >> "$fixture/docs/introduction.md"
+if ! grep -Fq '999 typed actions' "$fixture/docs/introduction.md"; then
+ printf 'FAIL: action-count mutation did not apply\n' >&2
exit 1
fi
-sed -i "s/${actions} typed actions/999 typed actions/" "$fixture/docs/introduction.md"
-assert_rejected 'action count that disagrees with the catalogue source'
+assert_rejected_with_diagnostic 'action count that disagrees with the catalogue source' \
+ 'docs/introduction.md' 'claims 999 typed actions, evidence says'
cp "$repo_root/docs/introduction.md" "$fixture/docs/introduction.md"
# An ActionSpec qualifier must not exempt a stale subset count from evidence.
-python3 - "$fixture/docs/developer-guide.md" <<'PYEOF'
-import re
-import sys
-from pathlib import Path
-path = Path(sys.argv[1])
-text = path.read_text(encoding="utf-8")
-updated, count = re.subn(
- r"[0-9]+ actions with an `ActionSpec`",
- "4 actions with an `ActionSpec`",
- text,
- count=1,
-)
-if count != 1:
- raise SystemExit("could not find the ActionSpec count in the fixture")
-path.write_text(updated, encoding="utf-8")
-PYEOF
+printf '\nThe executor holds 4 actions with an `ActionSpec`.\n' >> "$fixture/docs/developer-guide.md"
+if ! grep -Fq '4 actions with an `ActionSpec`' "$fixture/docs/developer-guide.md"; then
+ printf 'FAIL: ActionSpec mutation did not apply\n' >&2
+ exit 1
+fi
assert_rejected_with_diagnostic \
'ActionSpec count that disagrees with the generated table' \
'developer-guide.md' 'claims 4 actions with an ActionSpec' 'derived'
@@ -939,9 +992,10 @@ cp "$repo_root/docs/introduction.md" "$fixture/docs/introduction.md"
# ...and the counterpart: a real family size must still be quotable, or the guard
# would force every honest mention of the suite out of the docs.
-printf '\nThe atomic family is 54 stories.\n' >> "$fixture/docs/introduction.md"
-if ! "$checker" "$fixture" >/dev/null 2>&1; then
- printf 'FAIL: checker rejected a real family size (54 stories)\n' >&2
+printf '\nThe atomic family is %s stories.\n' \
+ "$(rendered stories_atomic 10)" >> "$fixture/docs/introduction.md"
+if ! output="$("$checker" "$fixture" 2>&1)"; then
+ printf 'FAIL: checker rejected a real, marked family size\n%s\n' "$output" >&2
exit 1
fi
cp "$repo_root/docs/introduction.md" "$fixture/docs/introduction.md"
@@ -1029,4 +1083,302 @@ grep -q '"rate_limited": 1' \
}
assert_rejected 'Validated tier backed by a replay with rate-limited stories'
+
+# ---------------------------------------------------------------------------
+# Figures are rendered, not typed (scripts/render_figures.py).
+#
+# Each refusal below is proved to bite by deleting its guard (see the mutation
+# notes in the commit that added it); a test here that stays green with its
+# guard removed is decorative.
+# ---------------------------------------------------------------------------
+renderer="$repo_root/scripts/render_figures.py"
+
+# The tier tests above left a mutated replay twin behind. Put every story run
+# back, and prove the whole fixture passes again, before attacking figures: a
+# rejection below must come from the figure under test.
+cp "$repo_root"/tests/evidence/story-runs/*.json "$fixture/tests/evidence/story-runs/"
+if ! output="$("$checker" "$fixture" 2>&1)"; then
+ printf 'FAIL: the fixture is rejected before the figure tests begin\n%s\n' "$output" >&2
+ exit 1
+fi
+
+# A stale marker fails, names the file and line, and shows both values. Only the
+# renderer can catch this one: the stored floor is not an exact count, so no
+# semantic check compares it.
+want_rust="$(rendered rust_tests 1000)"
+old_value="$(set_marker_value "$fixture/README.md" rust_tests '1,000+')"
+if [[ "$old_value" != "$want_rust" || "$want_rust" == '1,000+' ]]; then
+ printf 'FAIL: stale-marker mutation did not change the value (was %s, rendered %s)\n' \
+ "$old_value" "$want_rust" >&2
+ exit 1
+fi
+assert_rejected_with_diagnostic 'a stale figure marker' \
+ 'README.md:' "stale figure 'rust_tests'" "stored '1,000+'" "rendered '$want_rust'"
+restore_claim_file README.md
+
+# An unknown NAME fails and lists what is known.
+python3 - "$fixture/docs/architecture.md" <<'PYEOF'
+import sys
+from pathlib import Path
+path = Path(sys.argv[1])
+text = path.read_text(encoding="utf-8")
+if text.count("fig:typed_actions ") != 1:
+ raise SystemExit("expected exactly one typed_actions marker in architecture.md")
+path.write_text(text.replace("fig:typed_actions ", "fig:typed_actionz "), encoding="utf-8")
+PYEOF
+grep -Fq 'fig:typed_actionz' "$fixture/docs/architecture.md" || {
+ printf 'FAIL: unknown-NAME mutation did not apply\n' >&2
+ exit 1
+}
+assert_rejected_with_diagnostic 'an unknown figure NAME' \
+ 'docs/architecture.md:' "unknown figure NAME 'typed_actionz'" 'rust_tests'
+restore_claim_file docs/architecture.md
+
+# A figure outside a marker fails, even a rounded floor that no semantic check
+# could validate. This is the loophole: without it a hand-typed number creeps back.
+printf '\nThe suite has 2,000+ Rust tests.\n' >> "$fixture/README.md"
+assert_rejected_with_diagnostic 'a hand-typed floor outside a marker' \
+ 'README.md:' "figure '2,000+ Rust tests' is not inside a"
+restore_claim_file README.md
+printf '\nThe catalogue holds 190+ typed actions and 72 frontend tests.\n' >> "$fixture/README.md"
+assert_rejected_with_diagnostic 'hand-typed action and frontend figures' \
+ "figure '190+ typed actions'" "figure '72 frontend tests'"
+restore_claim_file README.md
+
+# The two README badges read the evidence file and carry no figure. They must be
+# present, or the pristine pass above says nothing about them.
+for badge in 'query=$.tests&label=Rust%20tests' 'query=$.frontend_tests&label=frontend%20tests'; do
+ if ! grep -Fq "$badge" "$fixture/README.md"; then
+ printf 'FAIL: README is missing the live badge %s\n' "$badge" >&2
+ exit 1
+ fi
+done
+
+# A marker inside a code fence fails; one inside an inline code span is syntax
+# documentation and does not.
+printf '\n```sh\n# %s\n```\n' "$want_rust" >> "$fixture/README.md"
+assert_rejected_with_diagnostic 'a figure marker inside a code fence' \
+ 'README.md:' 'figure marker inside a fenced code block'
+restore_claim_file README.md
+printf '\nWrite `VALUE` around a figure.\n' >> "$fixture/README.md"
+if ! output="$("$checker" "$fixture" 2>&1)"; then
+ printf 'FAIL: a marker shown in an inline code span was parsed\n%s\n' "$output" >&2
+ exit 1
+fi
+restore_claim_file README.md
+
+# Malformed markers: unbalanced and nested.
+printf '\nSee %s for the size.\n' "$want_rust" >> "$fixture/README.md"
+assert_rejected_with_diagnostic 'an unbalanced figure marker' \
+ 'README.md:' 'unbalanced figure marker'
+restore_claim_file README.md
+printf '\nSee 1.\n' >> "$fixture/README.md"
+assert_rejected_with_diagnostic 'a nested figure marker' \
+ 'README.md:' 'nested figure marker'
+restore_claim_file README.md
+
+# A doc that must state the test count loses its marker: dropping the figure
+# must not pass silently.
+python3 - "$fixture/docs/distro-support.md" <<'PYEOF'
+import re
+import sys
+from pathlib import Path
+path = Path(sys.argv[1])
+text = path.read_text(encoding="utf-8")
+updated, count = re.subn(r"[^<]* Rust tests", "unit tests", text)
+if count != 1:
+ raise SystemExit("expected one rust_tests marker in distro-support.md")
+path.write_text(updated, encoding="utf-8")
+PYEOF
+assert_rejected_with_diagnostic 'a required test-count doc that stopped stating it' \
+ 'docs/distro-support.md' 'does not state the measured baseline'
+restore_claim_file docs/distro-support.md
+
+# Rounding at the boundary, against fixture evidence: v = K*n renders n*K+, and
+# v = K*n - 1 renders (n-1)*K+, and v < K refuses. The fixture evidence is changed
+# through the real recorder's schema (only the `tests` field moves).
+set_tests() {
+ python3 - "$fixture/tests/evidence/workspace-tests.json" "$1" <<'PYEOF'
+import json
+import sys
+path = sys.argv[1]
+doc = json.loads(open(path, encoding="utf-8").read())
+doc["tests"] = int(sys.argv[2])
+open(path, "w", encoding="utf-8").write(json.dumps(doc, indent=2, sort_keys=True) + "\n")
+PYEOF
+}
+marker_of() { # file name -> the first marker's stored value
+ python3 - "$1" "$2" <<'PYEOF'
+import re
+import sys
+text = open(sys.argv[1], encoding="utf-8").read()
+match = re.search(r"([^<\n]*)