Skip to content

research: add SYM-ARCH-002A2 hierarchical statistics and power planning - #58

Draft
Tristan-Stoltz-ERC wants to merge 16 commits into
research/sym-arch-002a-core-v1from
research/sym-arch-002a2-hierarchical-power-v1
Draft

research: add SYM-ARCH-002A2 hierarchical statistics and power planning#58
Tristan-Stoltz-ERC wants to merge 16 commits into
research/sym-arch-002a-core-v1from
research/sym-arch-002a2-hierarchical-power-v1

Conversation

@Tristan-Stoltz-ERC

Copy link
Copy Markdown
Contributor

Summary

Adds the next focused SYM-ARCH-002A2 statistical-infrastructure tranche from issue #55.

This PR is deliberately stacked on #57 (research/sym-arch-002a-core-v1) and does not modify #57's frozen conceptual scope. It should remain draft/stacked until #57 lands, then be rebased/retargeted to main without changing the statistical contract.

No architecture winner/loser claim and no actual CONFIRM sample-size recommendation are produced by this PR.

Hierarchical inference

Adds experiment_statistics with:

  • NestedEnvironmentResult as the independent-world container;
  • PairedRunResult for candidate/control outcomes under the same nuisance realization;
  • unique environment and nuisance-run digests;
  • fail-closed duplicate identity checks;
  • equal-environment weighting regardless of nested run count;
  • environment-first + nested-run hierarchical bootstrap;
  • deterministic 95% percentile interval.

The nested interval is explicitly named percentile rather than silently reusing the existing one-level BCa label.

Prospective power

Adds a DEV-only Monte Carlo power planner that sizes future independent-environment counts against the same SESOI practical-effect gate introduced by #57.

Important safeguards:

  • only a valid exploratory DEV manifest may drive planning;
  • DEV result count must match the frozen environment-seed manifest;
  • DEV supplies residual/environment variability, not the assumed future effect;
  • planning_effect is separately frozen and must lie beyond the SESOI in the requested direction;
  • sampled DEV deltas are re-centered on planning_effect, reducing winner's-curse/tuning bias from the observed DEV mean;
  • residual_scale >= 1.0 supports conservative variance-inflation sensitivity analysis;
  • each simulated study succeeds only when the full environment-level BCa CI clears the practical SESOI margin;
  • power uncertainty itself is reported with a Wilson 95% interval;
  • minimum_environments is selected only when the lower power bound clears target and remains clear for every larger tested count.

If the tested count grid does not satisfy that sustained rule, the result is None; the target is not weakened after inspection.

Provenance

The power plan records:

  • digest of the exact DEV experiment manifest;
  • digest over DEV nested results and the full power configuration.

Changing data, planning effect, SESOI, variance inflation, counts, simulation settings, or seed changes the planning-input digest.

Tests

Focused tests cover:

  • equal environment weighting despite unequal nested-run counts;
  • deterministic hierarchical intervals;
  • duplicate environment rejection;
  • duplicate nuisance-run rejection;
  • deterministic power planning;
  • DEV-manifest provenance binding;
  • rejection of non-DEV planning;
  • manifest/result environment-count mismatch;
  • rejection of planning effects inside the SESOI margin;
  • bounded Monte Carlo power reporting;
  • no use of the observed DEV mean as the planning-effect center.

CI

Adds SYM-ARCH-002A2 Hierarchical Power, which supports both the current stacked base and a later main retarget. It runs:

  • rustfmt on A2 Rust paths;
  • focused experiment_statistics tests;
  • cargo check -p symthaea-psych-bench --lib.

Scientific boundary

This PR provides measurement/planning infrastructure only. It does not choose a planning effect, does not run a DEV campaign, does not open CONFIRM, and does not support any Symthaea capability claim.

The planning effect must be justified and frozen outside the code path. Multiple-comparison correction, procedural-world mutation tests, resource Pareto analysis, diagnostic state interventions, and stronger baselines remain separate later tranches.

Depends on #57. Tracks #55.

Copy link
Copy Markdown
Contributor Author

Static statistics audit — nuisance topology hardening

A post-implementation audit found a dependence-structure ambiguity worth fixing before A2 is allowed to inform CONFIRM sizing.

The original hierarchical code correctly treated environments as the independent scientific unit, but it modeled representation/learner/stream realizations as nuisance runs nested inside each environment. If the same nuisance realization is reused across multiple environments, those effects are crossed instead; independently resampling them inside each environment can distort uncertainty.

This PR now adds experiment_statistics_design with an explicit NuisanceTopology contract:

  • NestedIndependent: nuisance digests must be globally unique across environments.
  • CrossedShared: every environment must contain the same complete nuisance grid.
  • crossed uncertainty uses a two-way environment × nuisance bootstrap that resamples the nuisance identities jointly across environments.
  • the topology-checked v1 power entry point rejects crossed designs rather than silently using the nested prospective-power simulator.

A dedicated crossed prospective-power simulator remains future work and must be separately validated before a crossed CONFIRM design can use it.

The restriction and confirmatory freeze requirements are documented in docs/research/SYM_ARCH_002A2_NUISANCE_TOPOLOGY_V1.md and are included in the exact-head CI gate.

This is statistical-infrastructure hardening only; no architecture result or sample-size recommendation has been observed or changed.

Integration-only parent sync; no A2 scientific changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant