Skip to content

research: add SYM-ARCH-002A7 multiplicity-safe analysis plans - #66

Draft
Tristan-Stoltz-ERC wants to merge 6 commits into
research/sym-arch-002a2-hierarchical-power-v1from
research/sym-arch-002a7-multiplicity-v1
Draft

research: add SYM-ARCH-002A7 multiplicity-safe analysis plans#66
Tristan-Stoltz-ERC wants to merge 6 commits into
research/sym-arch-002a2-hierarchical-power-v1from
research/sym-arch-002a7-multiplicity-v1

Conversation

@Tristan-Stoltz-ERC

Copy link
Copy Markdown
Contributor

Summary

Adds SYM-ARCH-002A7 multiplicity-safe analysis-plan infrastructure as a draft stack directly on #58 / A2.

This PR adds no model, generates no raw p-values, consumes no CONFIRM data, and carries no architecture result. It closes the master plan's remaining multiple-comparison gap.

Frozen hypothesis families

MultiplicityPlan records:

  • versioned schema;
  • normalized family id;
  • family-wise alpha;
  • exact hypothesis ids;
  • frozen tail for every hypothesis (greater, less, two_sided);
  • primary/secondary role.

The plan receives an order-independent, domain-separated BLAKE3 digest. Changing family membership, role, tail, alpha, or family identity changes that digest.

Raw input integrity

RawHypothesisPValue carries both hypothesis id and the tail actually used by the separately preregistered raw test.

A7 fails closed unless:

  • the raw family matches the frozen hypothesis ids exactly;
  • ids are unique;
  • p-values are finite in [0,1];
  • each raw tail matches the frozen tail.

This prevents post-outcome one-sided/two-sided or direction switching from being hidden inside the multiplicity step.

A7 deliberately does not create raw p-values or certify the upstream test. Pairing unit, nuisance topology, test statistic, resampling method, and raw-test seed remain part of the separately frozen A2/experiment analysis contract.

Holm family-wise correction

apply_holm implements deterministic Holm step-down FWER control:

  1. raw p-values sorted ascending;
  2. deterministic tie-break by hypothesis id;
  3. multiply by hypotheses remaining;
  4. cap at 1;
  5. running maximum for monotone adjusted p-values;
  6. compare to frozen family alpha.

Output is canonical by hypothesis id, not significance rank.

This follows the same correction method already used by Muse's confirmatory analysis, but is independently implemented in psych-bench rather than coupling architecture research to Muse-specific types.

Simultaneous confidence intervals

A7 also exposes Bonferroni per-comparison alpha and the corresponding simultaneous confidence level for separately implemented interval estimators.

Holm-adjusted p-values and Bonferroni simultaneous intervals are separate reporting tools. The preregistration must freeze which inferential path supports each claim; they are not averaged into a score.

SESOI remains separate

Multiplicity controls family false-positive risk. SESOI/effect-size gates control practical importance. A statistically significant but practically tiny effect does not become an architecture win, and a large point estimate does not become confirmatory when multiplicity-safe uncertainty fails.

CI

Dedicated exact-head workflow gates:

  • rustfmt;
  • focused experiment_multiplicity tests;
  • full psych-bench library check on Rust 1.96.

Tests cover known Holm adjustment, order-independent plan identity, tail-sensitive identity, Bonferroni alpha, exact family matching, tail mismatch rejection, invalid p-values, and duplicate hypothesis rejection.

Wording ceiling

Merging A7 supports only:

Symthaea psych-bench can freeze claim-bearing hypothesis families and apply deterministic family-wise multiplicity control to separately preregistered inferential outputs.

It does not support a performance claim, certify an upstream raw test, replace SESOI/effect-size evidence, or permit post-hoc family splitting/tail switching.

Tracks #55. Depends on #58.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant