research: add SYM-ARCH-002A7 multiplicity-safe analysis plans - #66
Draft
Tristan-Stoltz-ERC wants to merge 6 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds SYM-ARCH-002A7 multiplicity-safe analysis-plan infrastructure as a draft stack directly on #58 / A2.
This PR adds no model, generates no raw p-values, consumes no CONFIRM data, and carries no architecture result. It closes the master plan's remaining multiple-comparison gap.
Frozen hypothesis families
MultiplicityPlanrecords:greater,less,two_sided);The plan receives an order-independent, domain-separated BLAKE3 digest. Changing family membership, role, tail, alpha, or family identity changes that digest.
Raw input integrity
RawHypothesisPValuecarries both hypothesis id and the tail actually used by the separately preregistered raw test.A7 fails closed unless:
[0,1];This prevents post-outcome one-sided/two-sided or direction switching from being hidden inside the multiplicity step.
A7 deliberately does not create raw p-values or certify the upstream test. Pairing unit, nuisance topology, test statistic, resampling method, and raw-test seed remain part of the separately frozen A2/experiment analysis contract.
Holm family-wise correction
apply_holmimplements deterministic Holm step-down FWER control:Output is canonical by hypothesis id, not significance rank.
This follows the same correction method already used by Muse's confirmatory analysis, but is independently implemented in psych-bench rather than coupling architecture research to Muse-specific types.
Simultaneous confidence intervals
A7 also exposes Bonferroni per-comparison alpha and the corresponding simultaneous confidence level for separately implemented interval estimators.
Holm-adjusted p-values and Bonferroni simultaneous intervals are separate reporting tools. The preregistration must freeze which inferential path supports each claim; they are not averaged into a score.
SESOI remains separate
Multiplicity controls family false-positive risk. SESOI/effect-size gates control practical importance. A statistically significant but practically tiny effect does not become an architecture win, and a large point estimate does not become confirmatory when multiplicity-safe uncertainty fails.
CI
Dedicated exact-head workflow gates:
experiment_multiplicitytests;Tests cover known Holm adjustment, order-independent plan identity, tail-sensitive identity, Bonferroni alpha, exact family matching, tail mismatch rejection, invalid p-values, and duplicate hypothesis rejection.
Wording ceiling
Merging A7 supports only:
It does not support a performance claim, certify an upstream raw test, replace SESOI/effect-size evidence, or permit post-hoc family splitting/tail switching.
Tracks #55. Depends on #58.