research: add SYM-ARCH-002A3 benchmark validity and mutation testing - #59
Draft
Tristan-Stoltz-ERC wants to merge 11 commits into
Draft
research: add SYM-ARCH-002A3 benchmark validity and mutation testing#59Tristan-Stoltz-ERC wants to merge 11 commits into
Tristan-Stoltz-ERC wants to merge 11 commits into
Conversation
This was referenced Aug 25, 2026
Contributor
Author
Static validity audit — support provenance omission is now detectedA validator audit found that The current head now requires at least one support provenance tag whenever declared-support enforcement is enabled. The mutation suite also adds The standard mutation suite is therefore now six corruptions:
This remains structural/oracle validity only; it does not widen A3's claim to universal shortcut freedom. |
Integration-only parent sync; no A3 scientific changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds SYM-ARCH-002A3 benchmark-integrity infrastructure from issue #55.
This is a draft sibling stack on #57 (
research/sym-arch-002a-core-v1), independent of #58's statistics work. It does not modify #57's frozen core and carries no architecture capability claim.The goal is simple: a generated benchmark must prove its own structural/oracle integrity—and prove that its validator detects known corruptions—before any model score is allowed to count as evidence.
Executable symbolic oracle
Adds an interpreter for the v1
RuleExprlanguage:symbolic_oracle_digestbindsTaskProgram.oracle_digestto the exact executableRuleExproracle used for label checking.Generated dataset contract
Adds:
ExampleRecordfeature maps;GeneratedTaskDatasetbound to the exact TaskProgram digest;Fail-closed validity checks
validate_generated_taskindependently reports:There is no scalar validity score: any violation makes the benchmark invalid.
Task-free leakage policy
BenchmarkValidityPolicy::task_free_strict()forbids explicit task/world/boundary keys by default and requires feature-disjoint splits, declared support tags, and both classes in both splits.Explicit-context experiments may relax the key policy explicitly; task identity is never silently permitted.
Mutation testing the scientific instrument
A3 deliberately corrupts a valid benchmark five ways:
mutation_detection_suitefirst requires the baseline benchmark to validate, then requires every mutated benchmark to become invalid.Wording boundary
Passing A3 v1 supports only:
It does not claim that every statistical shortcut is ruled out. Marginal-feature predictors, nearest-neighbor/lookup controls, shuffled-relation controls, semantic-equivalence checks, and temporal leakage probes remain follow-up construct-validity work.
Tests / CI
Focused tests cover:
Adds
SYM-ARCH-002A3 Benchmark Validitywith rustfmt, focused validity tests, and psych-bench library check.Scientific boundary
This PR contains no architecture comparison and no result claim. It is scientific instrumentation only.
Depends on #57. Can be reviewed in parallel with #58. Tracks #55.