Skip to content

research: add SYM-ARCH-002A4 adversarial construct-validity controls - #60

Draft
Tristan-Stoltz-ERC wants to merge 12 commits into
research/sym-arch-002a3-validity-v1from
research/sym-arch-002a4-construct-validity-v1
Draft

research: add SYM-ARCH-002A4 adversarial construct-validity controls#60
Tristan-Stoltz-ERC wants to merge 12 commits into
research/sym-arch-002a3-validity-v1from
research/sym-arch-002a4-construct-validity-v1

Conversation

@Tristan-Stoltz-ERC

Copy link
Copy Markdown
Contributor

Summary

Adds SYM-ARCH-002A4 adversarial construct-validity controls from issue #55.

This is a draft stack on #59 (research/sym-arch-002a3-validity-v1). It consumes A3's generated-dataset/oracle validity layer and asks a stricter question before any architecture score counts as evidence:

Can a scientifically trivial train-only shortcut solve the benchmark anyway?

If yes, the result is INCONCLUSIVE_BENCHMARK, not an architecture win/loss.

No architecture capability result is produced by this PR.

Train-only shortcut controls

A4 reports and gates:

  • training-majority predictor;
  • single-feature marginal predictor;
  • exact feature-assignment lookup;
  • deterministic categorical nearest neighbor;
  • prevalence-preserving shuffled-label single-feature controls;
  • prevalence-preserving shuffled-label nearest-neighbor controls;
  • executable symbolic-oracle positive control;
  • deterministic fair-chance sanity control.

The single-feature model chooses its feature using training accuracy only. Evaluation labels are used only for final scoring.

Fitted shortcut models consume learner-visible features only. example_id, support_tags, support metadata, and oracle outputs are excluded from fitted model inputs.

Finite-sample resolution gate

A4 computes the 95% Wilson upper accuracy bound for a fair Bernoulli classifier at the actual evaluation-set size.

If ordinary chance uncertainty can reach the tighter frozen shortcut ceiling, the benchmark is automatically INCONCLUSIVE_BENCHMARK and must increase evaluation support.

This intentionally catches coarse held-outs such as four-item evaluation splits before anyone treats a one-item difference as a precise architectural effect.

Fail-closed contract

A4 returns only:

  • passed;
  • inconclusive_benchmark.

It becomes inconclusive when:

  1. A3 structural/oracle validity fails;
  2. evaluation support cannot resolve the frozen ceiling;
  3. the executable oracle misses its frozen floor;
  4. majority, single-feature, exact lookup, or nearest neighbor reaches/exceeds the frozen shortcut ceiling;
  5. mean shuffled-label single-feature or nearest-neighbor accuracy reaches/exceeds the frozen shuffled-control ceiling.

A benchmark failure is never converted into a negative architecture claim.

Shuffled-relation controls

Training labels are deterministically permuted while preserving prevalence. At least four unique frozen shuffle seeds are required. A4 records both mean and maximum accuracy across shuffles; v1 gates on the mean to avoid making validity hinge on one lucky permutation.

Tests

Focused tests cover:

  • a clean XOR-style compositional benchmark passing low-complexity controls;
  • an injected single-feature label channel becoming inconclusive;
  • a four-item evaluation split failing the finite-resolution gate;
  • label-correlated support metadata leaving fitted shortcut scores unchanged;
  • prevalence preservation under deterministic label shuffling;
  • rejection of weak/duplicate shuffle contracts.

Threshold hygiene

The 0.80 ceilings used by unit tests are synthetic fixtures only. They are not scientific defaults.

Claim-bearing use must freeze before result inspection:

  • shortcut ceiling;
  • shuffled-label mean ceiling;
  • oracle floor;
  • chance seed;
  • complete shuffle-seed set;
  • A3 policy;
  • generator/version and seed manifest.

CI

Adds SYM-ARCH-002A4 Construct Validity with:

  • targeted rustfmt check;
  • focused construct-validity tests;
  • psych-bench library check.

The unrelated stale LoopTrialResult::cycle_reward fixture repair has already been propagated identically across #53/#57/#58/#59/A4 so this stack does not reintroduce that known compile failure. It does not appear in this PR's scientific diff.

Wording ceiling

A pass supports only:

The benchmark passed the implemented v1 structural/oracle and low-complexity shortcut controls under the frozen policy.

It does not establish that the benchmark is universally shortcut-free, and it does not establish Symthaea superiority.

Depends on #59. Tracks #55.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant