Skip to content

feat(scripts): kernel-level axiom sweep with committed regression baseline - #127

Open
alexanderlhicks wants to merge 3 commits into
mainfrom
alh/axiom-sweep
Open

feat(scripts): kernel-level axiom sweep with committed regression baseline#127
alexanderlhicks wants to merge 3 commits into
mainfrom
alh/axiom-sweep

Conversation

@alexanderlhicks

Copy link
Copy Markdown
Contributor

What

Adds lake exe axiomsweep: kernel-level axiom / sorry accounting for the whole library, with a committed regression baseline.

The tool loads the built .olean environment (the same data the kernel checked) and computes, for every declaration under the swept root modules, the set of axioms it transitively depends on — i.e. the #print axioms information, library-wide, in one pass. Because it reads elaborated data rather than source text, it sees exactly what the kernel accepted: private declarations, macro-generated declarations, and instances included; no regex heuristics.

Why

PolyFun feeds VCV-io and ArkLib, and its kernel-level state turns out to be pristine — 0 sorryAx, 0 non-standard axioms across all 8,006 declarations. This PR proves that and adds the regression gate that keeps it true, mirroring the same tool on the sibling repos. Reviews of downstream repos have repeatedly found theorems that silently inherit sorryAx through framework stubs; this makes that class of defect mechanical to catch.

How

  • scripts/AxiomSweep.lean (+ lean_exe axiomsweep in the lakefile): memoised traversal of the compiled environment; only imports core Lean, runtime-imports the built oleans, so it needs a completed lake build and adds no dependencies. Runs in seconds.
  • scripts/axiom_baseline.json: the currently-known sorryAx-tainted declarations (and any non-standard-axiom dependents). --check fails iff a declaration is tainted that the baseline does not list; closing gaps stays green and prints a nudge to shrink the baseline via --update-baseline.
  • CI: one report-only step after the build (::warning on regression). Flip to enforcing by deleting the || echo once the baseline has soaked for a few PRs.
  • Docs updated in the same PR.

Modes

lake exe axiomsweep                     # summary
lake exe axiomsweep --out report.json   # full per-declaration report (name, module, kind, line, axioms)
lake exe axiomsweep --check             # regression gate against scripts/axiom_baseline.json
lake exe axiomsweep --update-baseline   # refresh the baseline; commit the diff
lake exe axiomsweep --root Foo          # override the swept root modules (repeatable)

Baseline at this commit

  • 8,006 declarations across 224 PolyFun.* modules.
  • 0 sorryAx-tainted, 0 non-standard-axiom-tainted — fully kernel-clean.
  • Spot-checked against #print axioms (PFunctor.Display.responderCoalgebraEquiv_postcondition[Quot.sound]); ./scripts/validate.sh --axioms verified green end-to-end.

Notes

  • Same tool as ArkLib's alh/axiom-sweep branch (multi-root variant); the copies are intentionally self-contained per repo, mirroring how other scripts are shared across the org. If it proves its keep everywhere, extracting a shared home is a possible follow-up.
  • lean4export was considered and deliberately not used: it targets external kernel checking, has no source positions, and would add a per-toolchain pinned dependency; walking the environment yields the same information with line numbers and zero new deps.
  • The per-declaration report is not committed anywhere; only the baseline is tracked.

Adversarial review

Five independent adversarial reviewers (one per repo deployment) attacked the tool, wiring, and baselines before this PR was opened. Everything they confirmed is fixed in the follow-up commit on this branch:

  • Collector defect (HIGH, confirmed in 4 repos): the one-pass DFS finalized self-referencing constants (every inductive/ctor pair) prematurely, producing per-declaration rows that diverged from #print axioms in both directions, with constructible shapes where a new sorry passed --check. Fixed with a fixpoint-repair phase computing the true kernel closure (now strictly more accurate than #print axioms, whose own CollectAxioms wobbles inside mutual families). Post-fix rows spot-verified against ground truth; sorry-sets unchanged everywhere.
  • Axiom types now traversed (CollectAxioms parity); duplicate constNames rows deduped; native_decide axiom names normalized to their owning declaration (the ax_N_M counters are Elab.async/toolchain-volatile); --check/--update-baseline mutually exclusive; bare Lean.ofReduceBool/Lean.trustCompiler are never baselinable (floor).
  • CI steps now distinguish taint findings (report-only during soak) from infrastructure failures (always fail).
  • Known blind spots documented: structure-field defaults and examples never enter any environment walk (#print axioms included); unimported files are invisible — paired with each repo's import-completeness gate.

What survived attack unchanged: baseline determinism (byte-identical regeneration), all headline numbers, spot-checks vs #print axioms, gate exit-code semantics, and the CI/step placement.

🤖 Generated with Claude Code

alexanderlhicks and others added 3 commits August 10, 2026 12:40
…eline

Add `lake exe axiomsweep`: walks the compiled environment and computes,
for every PolyFun.* declaration, its transitive axiom dependencies —
the #print axioms information, library-wide, in one pass. Reads
elaborated .olean data, so private and macro-generated declarations are
included and no source heuristics are involved.

Baseline at this commit: 8006 declarations across 224 modules, 0
sorryAx-tainted, 0 non-standard-axiom-tainted — the library is fully
kernel-clean, and --check now keeps it that way (fails iff a declaration
is tainted that scripts/axiom_baseline.json does not list).

Wire-up: ./scripts/validate.sh --axioms (enforcing, opt-in; verified
green end-to-end) and a report-only CI step in ci.yml;
docs/wiki/quickstart.md documents the workflow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, docs

Collector: two-phase DFS + fixpoint repair. The one-pass DFS finalized
self-referencing constants (every inductive/ctor pair) prematurely and
memoized the wrong result for all later roots — reviews confirmed rows
diverging from #print axioms (both directions) and constructible shapes
where a new sorry passed --check. The repair pass re-derives every set
in finalization order until stable: the least fixpoint = true kernel
closure, strictly more accurate than #print axioms inside mutual
families. Also: axiom *types* traversed (CollectAxioms parity),
duplicate constNames rows deduped, native trust-axiom names normalized
to their owner (ax_N_M counters are Elab.async/toolchain-volatile),
--check/--update-baseline mutually exclusive, unknown --root fails
gracefully, nonstandard shrinkage detected, bare
Lean.ofReduceBool/Lean.trustCompiler never baselinable (floor).

Verified post-fix: sorry-set unchanged; previously-wrong rows now match
#print axioms ground truth; regeneration deterministic; gate exit codes
intact. Known blind spots documented (structure-field defaults and
examples never enter any environment walk; unimported files — paired
with the repo's import-completeness gate).

CI: infrastructure failures (exit != 1) now fail the step; only taint
findings are report-only during the soak.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

PolyFun specifics: baseline unchanged (8006/224, 0/0 — reviewer's 14
attribution-divergent rows involved only standard axioms); --axioms
documented in the AGENTS.md Building section; --root on a glob-based lib
(PolyFunTest) now errors cleanly instead of crashing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants