Skip to content

research: add SYM-ARCH-002A3 benchmark validity and mutation testing - #59

Draft
Tristan-Stoltz-ERC wants to merge 11 commits into
research/sym-arch-002a-core-v1from
research/sym-arch-002a3-validity-v1
Draft

research: add SYM-ARCH-002A3 benchmark validity and mutation testing#59
Tristan-Stoltz-ERC wants to merge 11 commits into
research/sym-arch-002a-core-v1from
research/sym-arch-002a3-validity-v1

Conversation

@Tristan-Stoltz-ERC

Copy link
Copy Markdown
Contributor

Summary

Adds SYM-ARCH-002A3 benchmark-integrity infrastructure from issue #55.

This is a draft sibling stack on #57 (research/sym-arch-002a-core-v1), independent of #58's statistics work. It does not modify #57's frozen core and carries no architecture capability claim.

The goal is simple: a generated benchmark must prove its own structural/oracle integrity—and prove that its validator detects known corruptions—before any model score is allowed to count as evidence.

Executable symbolic oracle

Adds an interpreter for the v1 RuleExpr language:

  • feature equality/inequality;
  • parity relations;
  • recursive NOT/AND/OR/XOR composition;
  • fail-closed missing-feature behavior.

symbolic_oracle_digest binds TaskProgram.oracle_digest to the exact executable RuleExpr oracle used for label checking.

Generated dataset contract

Adds:

  • deterministic ExampleRecord feature maps;
  • explicit support tags and expected labels;
  • GeneratedTaskDataset bound to the exact TaskProgram digest;
  • canonical set-style dataset hashing;
  • feature-only example digests so a train/eval duplicate cannot hide behind a new id/label.

Fail-closed validity checks

validate_generated_task independently reports:

  • invalid TaskProgram;
  • TaskProgram/dataset digest mismatch;
  • executable-oracle digest mismatch;
  • empty splits;
  • invalid examples;
  • duplicate example ids;
  • feature-identical train/eval leakage;
  • forbidden task/world/boundary identity features;
  • undeclared support tags;
  • labels inconsistent with the symbolic oracle;
  • declared vs observed class-count mismatch;
  • split class degeneracy when required.

There is no scalar validity score: any violation makes the benchmark invalid.

Task-free leakage policy

BenchmarkValidityPolicy::task_free_strict() forbids explicit task/world/boundary keys by default and requires feature-disjoint splits, declared support tags, and both classes in both splits.

Explicit-context experiments may relax the key policy explicitly; task identity is never silently permitted.

Mutation testing the scientific instrument

A3 deliberately corrupts a valid benchmark five ways:

  1. flip an evaluation label;
  2. leak a training feature assignment into evaluation;
  3. corrupt the TaskProgram digest;
  4. inject a forbidden task-id feature;
  5. inject undeclared evaluation support.

mutation_detection_suite first requires the baseline benchmark to validate, then requires every mutated benchmark to become invalid.

Wording boundary

Passing A3 v1 supports only:

structural/oracle benchmark integrity passed

It does not claim that every statistical shortcut is ruled out. Marginal-feature predictors, nearest-neighbor/lookup controls, shuffled-relation controls, semantic-equivalence checks, and temporal leakage probes remain follow-up construct-validity work.

Tests / CI

Focused tests cover:

  • nested boolean oracle evaluation;
  • executable-oracle digest binding;
  • order-insensitive dataset identity;
  • feature-level split leakage;
  • forbidden task identity;
  • declared class counts;
  • all five benchmark mutations.

Adds SYM-ARCH-002A3 Benchmark Validity with rustfmt, focused validity tests, and psych-bench library check.

Scientific boundary

This PR contains no architecture comparison and no result claim. It is scientific instrumentation only.

Depends on #57. Can be reviewed in parallel with #58. Tracks #55.

Copy link
Copy Markdown
Contributor Author

Static validity audit — support provenance omission is now detected

A validator audit found that require_declared_support_tags=true checked every present tag against the split contract but allowed an example with zero support tags to pass vacuously.

The current head now requires at least one support provenance tag whenever declared-support enforcement is enabled. The mutation suite also adds RemoveFirstEvalSupport, so A3 attacks omission as well as an injected undeclared tag.

The standard mutation suite is therefore now six corruptions:

  1. wrong eval label;
  2. train→eval feature leakage;
  3. corrupt program digest;
  4. hidden task id;
  5. undeclared eval support;
  6. missing eval support provenance.

This remains structural/oracle validity only; it does not widen A3's claim to universal shortcut freedom.

Integration-only parent sync; no A3 scientific changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant