research: SYM-ARCH-002 stronger continual-compositional validation plan

[Tristan-Stoltz-ERC]

Goal

Design SYM-ARCH-002 as a separate preregistered follow-up to SYM-ARCH-001. Do not alter PR #53's scientific configuration or decision rule after its result is observed.

002 should answer a stronger question:

Which Symthaea mechanisms, if any, produce a reproducible advantage under task-free, compositionally novel, resource-normalized continual learning against stronger controls?

Why a separate experiment

SYM-ARCH-001 intentionally tests a narrow mechanism-level setting with explicit world context, fixed-step HDC-LTC evolution, four simple relational worlds, and lightweight baselines. That is appropriate for a first discrimination test, but it leaves several important validity gaps:

• explicit world/task identity is injected into each input;
• fixed dt=0.05 under-tests the claimed continuous-time advantage;
• the SSM control is frozen rather than trainable;
• no strong fixed-random-feature/RanDumb-style baseline exists;
• compositionality is limited to held-out first-order factor combinations;
• statistics compare candidate means against the best control mean rather than paired per-seed deltas;
• only final/peak summaries are recorded rather than a complete task-by-time performance matrix;
• resource normalization is limited to observations/dimension/wall time.

002 tranche structure

002A — Evaluation core

Build reusable continual-learning evaluation infrastructure before adding new candidate mechanisms.

Required outputs:

• full performance matrix R[t_train][t_eval];
• average incremental accuracy;
• backward transfer;
• forward transfer;
• forgetting;
• examples-to-criterion;
• area under online accuracy curve;
• paired per-seed candidate-minus-control deltas;
• bootstrap 95% CIs on paired deltas;
• simultaneous/multiple-comparison-safe reporting;
• trainable parameter count;
• persistent state bytes;
• replay bytes;
• update latency p50/p95;
• inference latency p50/p95;
• throughput;
• peak RSS where available.

No architecture winner/loser claim from 002A itself.

002B — Baseline ladder

Add stronger controls under matched data and update opportunities:

1. majority/chance sanity controls;
2. exact lookup/memorization control;
3. nearest-neighbor control;
4. fixed random features + linear/RLS readout (RanDumb-style control);
5. vanilla HDC + matched readout;
6. online recursive least squares / forward-only adaptation;
7. bounded reservoir replay baseline;
8. small GRU or LSTM;
9. trainable diagonal/selective SSM;
10. small Mamba-class baseline where dependency cost is acceptable;
11. HDC-LTC frozen;
12. HDC-LTC + plasticity candidate.

The purpose is not to maximize baseline complexity; it is to prevent an HDC-LTC win from being explainable by a weak control set.

002C — Factorial Symthaea ablations

Factor the candidate instead of treating it as one monolith.

Axes:

• representation: raw/random vs HDC;
• dynamics: none vs fixed SSM vs LTC;
• time constant: fixed tau vs adaptive tau;
• state: reset-per-item vs persistent state;
• plasticity: none vs Hebbian vs available contrastive/local rule;
• associative memory: off vs on;
• readout: prototype vs RLS where compatible.

Primary analysis: main effects and interactions. A positive full-candidate result is insufficient unless at least one component/interaction can be identified as causally responsible.

002D — Context and drift ladder

Three context conditions:

1. explicit_context: world/task identity supplied (001-like control);
2. latent_context: context must be inferred from recent observations;
3. task_free: no task identity or explicit boundary signal.

Drift families:

• abrupt;
• gradual;
• recurring;
• cyclic;
• stochastic hazard / unsignaled rule switches.

No benchmark component may expose future boundaries to the learner.

002E — Compositionality ladder

Measure distinct forms of compositional generalization:

• systematicity: unseen combinations of known factors/relations;
• productivity: longer/deeper compositions than those seen in training;
• substitutivity: replace a known primitive with an equivalent/new-role primitive and preserve relational behavior;
• unseen role swaps;
• three-factor compositions;
• unseen compositions of learned transformations;
• relations-of-relations.

Increase factor cardinality beyond the 4x4 setting to avoid coarse four-item held-out accuracies.

Include construct-validity controls:

• marginal-factor predictor;
• nearest neighbor;
• exact lookup;
• shuffled-relation negative control;
• symbolic oracle / known-answer upper bound.

If negative controls behave unexpectedly or the oracle fails, return INCONCLUSIVE rather than scoring the architecture.

002F — Continuous-time stress test

This should directly test the claimed reason to use LTC/CfC dynamics.

Present the same semantic stream under:

• uniform cadence;
• bursty cadence;
• long gaps;
• dropped/skipped observations;
• temporal dilation/compression;
• jittered timestamps;
• mixed-rate sensors.

Compare fixed-step and timestamp-aware evolution. Keep semantic content identical while changing only timing.

Primary question:

Does adaptive continuous-time state preserve retention/adaptation under timing distortions better than matched discrete/fixed-step baselines?

Statistical design

Use paired seeds and identical streams across all systems.

For each primary metric, report per-seed candidate-minus-control deltas and bootstrap confidence intervals. Prefer one preregistered primary comparator per claim; secondary comparisons should use simultaneous intervals or family-wise/FDR correction.

Do not classify on point estimates alone.

Suggested claim gate for any architecture-level positive result:

• construct-validity controls all pass;
• candidate paired CI clears the preregistered practical margin on >=2 primary phenomena;
• no primary metric shows a practically meaningful regression;
• advantage survives the relevant mechanism ablation;
• advantage survives at least one harder context condition beyond explicit_context;
• advantage is not explained by random-feature or replay baselines;
• resource use remains within the preregistered comparison budget.

Otherwise classify as MIXED, NULL, NEGATIVE, or INCONCLUSIVE as appropriate.

External-validation ladder

After synthetic mechanism discrimination is stable, add externally recognizable tasks rather than immediately jumping to general language benchmarks:

• continual compositional NLI / primitive-to-composition tasks;
• compositional-ARC-style unseen transformation compositions;
• established online continual-learning datasets with task-free streams;
• a real sequential control/sensor task with irregular timestamps.

The goal is to bridge from mechanism evidence to externally legible capability evidence.

Research anchors

Recent work motivating this design includes:

• C2Gen NLI: continual acquisition of primitives + unseen compositional inference; forgetting remains a central challenge.
• Continual Learning Bench 1.0 (2026): realistic dependent task sequences where systems are expected to change during evaluation.
• CompIL (AAAI 2026): composition-incremental learning and progressively arriving novel compositions.
• task-free online continual learning work (CVPR/UAI 2026): dynamic streams without explicit task boundaries.
• continual-compositional evaluation frameworks distinguishing systematicity, productivity, and substitutivity.
• fixed-random-representation/RanDumb results showing that random features can be an unexpectedly strong continual-learning baseline.

Ordering / stop rules

Implement in this order:

1. 002A evaluation core;
2. 002B baseline ladder;
3. 002C ablations;
4. 002D context/drift;
5. 002E compositionality;
6. 002F continuous-time stress;
7. external-validation tranche.

Stop and fix the benchmark before proceeding if construct-validity controls fail. Stop adding architecture complexity if a simpler baseline explains the observed gain.

Relationship to PR #53

PR #53 / SYM-ARCH-001 remains frozen as the first preregistered discrimination experiment. 002 should consume its result as evidence, not rewrite its thresholds or interpretation.

[Tristan-Stoltz-ERC]

Further refinement — confirmation hygiene, causal identification, and semantic timescales

The next pass should add three cross-cutting safeguards/capabilities to the 002 program.

1. Confirmation hygiene: separate development, confirmation, and replication

A continually iterated benchmark can itself become overfit. Treat task/seed reuse as a source of adaptive bias.

Define three non-overlapping stream namespaces:

• DEV: visible, reusable engineering streams. May be used for debugging, profiling, hyperparameter search, architecture changes, and baseline bring-up. Results are explicitly exploratory.
• CONFIRM: untouched preregistered streams. The first successful run on the frozen preregistration commit is the primary confirmatory result. Architecture/hyperparameter changes after observing these results create a new experiment version.
• REPL: second independent stream family used only after the confirmatory result is frozen. Its role is replication, not rescue.

Required evidence metadata per run:

• experiment version;
• code commit;
• preregistration commit/hash;
• stream namespace and generator version;
• seed manifest hash;
• whether any prior result from that namespace had been observed;
• tuning status (exploratory, confirmatory-first-use, replication, post-hoc).

A rerun caused purely by infrastructure/compiler failure may retain confirmatory status only if no behavioral metrics were produced. Once primary metrics were visible, the namespace is considered observed.

Prefer procedural task generation over a small hand-authored rule set. Freeze the generator first, then freeze the confirmatory seed manifest. This lets the project iterate rapidly on DEV without silently converting CONFIRM into another tuning set.

2. Add a genuine causal-intervention ladder

Contingency reversal is useful but does not distinguish correlation tracking from causal-model adaptation.

Add synthetic structural causal models (SCMs) where ground-truth graph/mechanisms are known. Separate four changes that ordinary accuracy can conflate:

1. covariate drift — P(X) changes while causal mechanisms remain fixed;
2. label/reward drift — outcome mapping changes without changing physical transition structure;
3. mechanism drift — exactly one structural equation/edge changes;
4. intervention — do(X=x) replaces the natural generating mechanism for one variable.

Include observational training distributions with deliberately strong spurious correlations/confounding, then intervention distributions that break those correlations.

Primary causal metrics:

• interventional prediction accuracy;
• causal-direction discrimination;
• spurious-correlation rejection;
• changed-mechanism localization;
• adaptation latency after a mechanism intervention;
• preservation of unaffected mechanisms;
• recovery speed when an old causal regime recurs.

Key construct-validity controls:

• correlational predictor that should fail under intervention;
• oracle SCM reasoner that must pass;
• shuffled-graph negative control;
• intervention-label shuffle;
• observational-only condition.

A particularly strong result would be localized repair: after one causal mechanism changes, the learner rapidly updates behavior involving that mechanism while predictions involving invariant mechanisms remain stable. Broad forgetting followed by relearning should not receive the same credit.

3. Semantic-timescale / stability-plasticity experiment

Test the factorized-timescale hypothesis directly rather than assuming adaptive tau is beneficial.

Create streams with simultaneously present information at different true timescales:

• long-lived: object identity, stable relational rules, ethical/safety constraints or invariant physical laws;
• medium-lived: task strategy, partner policy, environmental regime;
• short-lived: immediate sensory cue, temporary goal, transient noise.

Compare:

1. no temporal state;
2. one fixed shared tau;
3. one globally adaptive tau;
4. multiple fixed timescale groups;
5. adaptive group-specific tau_g;
6. adaptive tau_g + associative memory/plasticity.

Stress by changing short-timescale context repeatedly while preserving long-timescale rules, then invert the experiment: change one long-timescale rule while adding large amounts of transient distractor activity.

Measure:

• fast-context adaptation latency;
• stable-rule retention;
• cross-timescale interference;
• recovery after transient perturbation;
• false consolidation rate (transient event incorrectly becoming persistent);
• false forgetting rate (stable information overwritten by transient events);
• per-group state drift;
• per-group effective tau trajectories.

The desired evidence is a double dissociation: short-timescale groups adapt rapidly without corrupting long-lived representations, while long-timescale groups remain changeable when sustained evidence genuinely indicates a regime change.

4. Add Pareto and simplicity gates

Do not collapse accuracy, memory, compute, and adaptation into one arbitrary scalar score.

Report Pareto fronts over at least:

• continual/compositional performance;
• forgetting;
• examples-to-criterion;
• update throughput;
• inference latency;
• peak/persistent memory;
• trainable parameters/state bytes.

Run at several matched budgets where practical:

• equal observations;
• equal trainable-parameter budget;
• equal persistent-memory/state-byte budget;
• approximately equal update-compute budget.

If two systems are practically equivalent on capability and one is materially simpler/cheaper, the simpler system wins by default. Complexity must earn itself.

5. Add scaling curves, not just one dimension

After the mechanism and task design validate on DEV, sweep representation/state scale (for example 64/128/256/512/1024/2048 where feasible) and data budget.

Fit/plot capability vs:

• HDC dimension;
• trainable parameters;
• examples seen;
• state bytes;
• compute/update.

Important questions:

• Does the candidate advantage grow, shrink, or disappear with dimension?
• Is any apparent gain simply a high-dimensional random-feature effect?
• Is there a crossover point where HDC-LTC becomes more sample-efficient than neural controls?
• Does the mechanism saturate early?

Do not use the scaling sweep to choose the confirmatory operating point after seeing CONFIRM results. Choose/freeze the confirmatory budget using DEV only.

6. Suggested expanded outcome taxonomy

In addition to PASS/MIXED/NULL/NEGATIVE/INCONCLUSIVE, record a machine-readable failure/qualification reason, e.g.:

• CONTROL_FAILURE
• ORACLE_FAILURE
• RANDOM_FEATURE_EXPLAINS_GAIN
• REPLAY_EXPLAINS_GAIN
• REPRESENTATION_ADVANTAGE
• TEMPORAL_ADVANTAGE
• PLASTICITY_ADVANTAGE
• MEMORY_ADVANTAGE
• INTERACTION_ADVANTAGE
• CAUSAL_GENERALIZATION_NOT_DEMONSTRATED
• TASK_FREE_ADAPTATION_NOT_DEMONSTRATED
• RESOURCE_REGRESSION
• CONFIRMATORY_STREAM_PREVIOUSLY_OBSERVED

This keeps a null/negative outcome architecturally useful instead of reducing it to a single label.

Revised ordering

1. 002A evaluation + evidence schema + DEV/CONFIRM/REPL separation;
2. 002B strong/simple baseline ladder;
3. 002C factorial mechanism ablations;
4. 002D task-free/context drift;
5. 002E compositionality taxonomy;
6. 002F irregular-time stress;
7. 002G causal-intervention/localized-repair suite;
8. 002H semantic-timescale stability/plasticity suite;
9. independent replication + externally recognizable tasks.

The two strongest possible end-state claims are now intentionally narrower and more defensible:

a specific Symthaea mechanism causes a reproducible capability gain under the conditions theory predicts, against strong alternative explanations;

and

that gain replicates on an untouched task stream under a matched resource budget.

Anything weaker should be reported at the level actually supported by the evidence.

[Tristan-Stoltz-ERC]

Further refinement — avoid pseudoreplication; make the task family the unit of generalization

A critical design improvement for 002: do not let seed mean only "different random HDC vectors/order over the same small set of hand-authored rules." That estimates run variance, but it does not establish that an architectural effect generalizes across problems.

Factor random variation into separate sources

Use independent seed namespaces for:

• environment_seed — generates a genuinely different relational/causal task program/world family;
• representation_seed — changes HDC/random-feature basis while holding the semantic task fixed;
• learner_seed — changes trainable model initialization/stochastic learner state;
• stream_seed — changes presentation/order/timing while holding the task fixed.

Where compute permits, use a crossed or partially crossed design so the same generated environment is evaluated under multiple representation/learner/order seeds, and multiple independent environments are evaluated by every architecture.

This lets us answer different questions explicitly:

• environment variance: does the claimed advantage generalize to new problems/rules?
• representation variance: is it robust to arbitrary HDC basis choice?
• initialization variance: is the learner stable?
• stream variance: does ordering/cadence drive the effect?

Procedural task-program generator

Replace the tiny fixed rule list in the confirmatory suite with a frozen grammar capable of generating many valid tasks while retaining exact ground truth.

Candidate primitive grammar:

• equality / inequality;
• modular relations;
• parity relations;
• ordered comparisons;
• role-sensitive relations;
• conjunction/disjunction/negation;
• permutation and composition of primitive transformations;
• small relational chains;
• SCM mechanisms for the causal tranche.

Each generated task must carry machine-verifiable metadata:

• task AST/program;
• complexity/depth;
• class balance;
• train/test support overlap;
• held-out composition class;
• exact oracle answers;
• transformation/invariance group where applicable.

Generator validity tests should reject degenerate tasks before any learner sees them: trivial constant labels, duplicate-equivalent programs, accidental train/test leakage, class imbalance outside tolerance, or a baseline/oracle ladder that does not meet its expected behavior.

Hierarchical reporting

Do not report only one CI over a flat list of runs. Preserve results grouped by generated environment/task family.

At minimum report:

• macro-average over environments (each environment equal weight);
• within-environment seed variance;
• between-environment variance;
• paired candidate-minus-control delta per environment;
• fraction of independent environments where the candidate clears the practical margin.

For stronger confirmatory analysis, use a hierarchical/bootstrap procedure that resamples environments first and within-environment runs second. This makes the inferential target "new tasks from this generator family," rather than "new random initializations on the same four tasks."

Claim gate

A positive architecture claim should require both:

1. the aggregate hierarchical CI clearing the practical margin; and
2. the advantage not being driven by only one or two generated task families.

Suggested robustness gate: the candidate should have non-negative paired performance on a preregistered supermajority of independent environments, in addition to the aggregate effect criterion.

Power / sample-size rule

Choose confirmatory environment count from DEV variance of paired environment-level deltas, not arbitrary run count. Freeze:

• practical effect threshold;
• target power (prefer 0.90 for the primary claim if affordable);
• family-wise alpha / correction strategy;
• maximum confirmatory environment count;
• runs per environment.

The confirmatory sample size may be computed from DEV/pilot variance, but must be frozen before CONFIRM streams are generated/run. Do not increase n because CONFIRM happened to miss significance.

This changes 002 from "many seeds on a benchmark" into a much stronger question:

does the mechanism advantage generalize across independently generated relational/causal worlds, arbitrary representation bases, and stream realizations?

[Tristan-Stoltz-ERC]

Implementation constraint — extend the existing psych-bench statistics layer, do not fork it

Repository review shows symthaea-psych-bench already contains reusable statistical infrastructure:

• MetricValue::from_samples_bootstrap with deterministic BCa bootstrap CIs;
• t-based small-sample CIs;
• Cohen's d helpers;
• Spearman correlation matrices;
• split-half reliability;
• Campbell-Fiske-style convergent/discriminant construct-validity analysis.

002A should therefore extend harness::analysis / harness::report rather than implementing a private statistics stack in benchmarks::architecture.

Add reusable primitives there for:

• paired deltas and paired BCa CIs;
• hierarchical bootstrap (environment first, within-environment run second);
• continual-learning R[train_phase][eval_task] matrix metrics (AIA/BWT/FWT/forgetting);
• practical-equivalence / non-inferiority margins;
• multiple-comparison metadata/correction;
• Pareto-front extraction;
• latency/throughput quantiles;
• variance decomposition by environment/representation/learner/stream seed.

The architecture benchmark should primarily generate trials and observations; the harness should own statistical interpretation. This keeps future continual-learning, morphogenesis, robotics, and other adaptive benchmarks able to reuse the same claim-integrity machinery.

[Tristan-Stoltz-ERC]

Further refinement — power, mechanism observability, and generator-level OOD

This pass adds three safeguards/capabilities that should be folded into 002A/002C rather than creating another parallel benchmark stack.

1. Prospective power and smallest-effect-of-interest gate

Do not choose the number of confirmatory environments by convention (16 seeds, 32 runs, etc.). Use DEV-only data to estimate the number of independent environments required to detect the smallest effect size that would actually justify architectural complexity.

Define before CONFIRM:

• one or more primary paired deltas;
• a smallest effect size of interest (SESOI), preferably in practical units (e.g. +0.05 absolute accuracy, -20% adaptation latency, +X retained tasks per MiB) rather than standardized effect alone;
• target power (e.g. 0.8 or 0.9);
• alpha / simultaneous-interval policy;
• number of independent environment programs;
• number of nested representation/learner/order replicates per environment.

Estimate power with a hierarchical/bootstrap simulation using DEV effect distributions. Freeze the resulting CONFIRM sample size and manifest before first use. Do not repeatedly peek at confirmatory results and add environments until significance appears; if sequential analysis is ever used, preregister an explicit alpha-spending or confidence-sequence rule.

A PASS requires the paired CI to clear the SESOI, not merely zero. A narrow CI entirely inside the practical-equivalence region should support an explicit EQUIVALENT/NO_PRACTICAL_ADVANTAGE qualification rather than an ambiguous NULL.

2. Mechanism observability + causal state interventions

Behavioral ablations tell us that removing a component hurts; they do not by themselves show what information that component carries or whether that internal state causally mediates the behavior.

Instrument the candidate and compatible controls with a common diagnostic trace interface. Capture, at minimum where available:

• representation/state vector snapshots;
• per-group effective tau / gates;
• Hebbian update magnitude and direction;
• associative-memory key/query/retrieval similarity;
• prediction margin/confidence;
• change-detection signal;
• state norm and update norm;
• component-level wall/update time.

Add representation diagnostics:

• within-concept vs between-concept similarity;
• role/factor decodability with frozen linear probes trained only on DEV;
• relational decodability;
• representation drift across task phases;
• subspace/geometry similarity (e.g. CKA/RSA-style comparison where mathematically appropriate);
• invariant-mechanism drift vs changed-mechanism drift in SCM tasks;
• memory retrieval precision/false-association rate.

Then add causal internal interventions rather than relying only on probes:

1. run a clean trajectory;
2. run a matched corrupted/counterfactual trajectory;
3. replace one candidate component/state (HDC binding result, liquid state, tau_g, memory retrieval, etc.) in the corrupted run with its clean counterpart;
4. measure recovery of the correct behavioral output;
5. perform the reverse patch (corrupt one component in the clean trajectory) and measure degradation.

This is analogous in spirit to activation/interchange patching: a useful internal variable should not merely correlate with success; intervening on it should causally alter the relevant capability. Avoid overclaiming if multiple components redundantly encode the same information.

Desired evidence examples:

• patching the long-timescale state restores stable-rule retention but not immediate context response;
• patching short-timescale state restores rapid context adaptation but not long-term memory;
• patching HDC relational state restores compositional inference while temporal adaptation is unchanged;
• patching associative-memory retrieval restores previously learned facts without repairing causal-rule adaptation.

That would provide a mechanistic double dissociation, not just an ablation table.

3. Generator-level OOD splits, not only new RNG worlds

Procedurally generating new environments is necessary but still insufficient if CONFIRM environments come from exactly the same grammar productions and complexity range as DEV.

The task generator should expose a typed notion of program structure and support OOD partitions along explicit axes:

• unseen operator combinations;
• unseen relation graph motifs;
• greater composition depth;
• larger arity/cardinality;
• longer dependency horizon;
• different causal graph topology;
• denser/sparser confounding;
• new drift schedule family;
• novel timescale ratios;
• new intervention targets;
• unseen mixtures of previously learned primitives.

Use at least three generalization tiers:

• IID_PROGRAM: new environment programs from the same grammar/complexity support;
• OOD_COMPOSITION: same primitives, unseen combinations/topologies/depths;
• OOD_FAMILY: one or more generator dimensions held outside DEV support.

The strongest architecture claim should require survival beyond IID_PROGRAM.

4. Build a typed benchmark-program IR/compiler

Rather than hand-author every SYM-ARCH task, introduce a small reusable task-program IR in symthaea-psych-bench from which relational, temporal and causal streams can be compiled.

A program should explicitly declare:

• primitives/roles;
• relation or structural equations;
• context visibility (explicit, latent, task_free);
• timing process;
• drift/intervention schedule;
• observation/noise model;
• oracle answer function;
• metamorphic invariances and required counterfactual changes;
• difficulty/complexity metadata.

Compilation should produce:

• deterministic stream given the four seed namespaces;
• oracle labels/causal answers;
• construct-validity controls;
• a content hash / canonical serialization for provenance;
• automatic metamorphic tests.

This would turn the generator into a research instrument and make environment_seed genuinely mean a different problem rather than merely a different vector basis.

5. Preflight benchmark-validity gate

Before any candidate-vs-baseline result can count, the generated environment family must pass DEV preflight:

• oracle >= preregistered near-perfect threshold;
• chance/marginal controls near expected floor;
• lookup succeeds on seen items but fails appropriately on true OOD items;
• shuffled-relation/graph controls fail;
• no accidental label imbalance or factor leakage;
• no trivial single-feature predictor clears the target task;
• candidate and strongest baseline are neither universally at floor nor ceiling;
• metamorphic invariances/counterfactuals behave as specified;
• task difficulty spans a useful range rather than one narrow regime.

If these fail, classify the benchmark INVALID/INCONCLUSIVE and fix the generator on DEV before touching CONFIRM.

6. Behavioral fingerprints, not accuracy alone

For matched test items, preserve item-level outcomes and compare systems by which errors they make, not only how many. Report:

• paired per-item disagreement matrix;
• error consistency / agreement with oracle failure modes where meaningful;
• calibration/Brier score for probabilistic systems;
• false-change-alarm rate;
• miss rate for real change points;
• overreaction to transient noise;
• recovery trajectory shape after a switch;
• confidence before/after intervention;
• error clustering by task-program feature (depth, topology, horizon, timescale).

This can reveal cases where two systems have the same mean accuracy but qualitatively different cognitive failure modes.

Revised implementation implication

002A should now contain four reusable pieces:

1. experiment_registry — DEV/CONFIRM/REPL manifests + provenance;
2. task_program — typed IR/compiler + oracle + OOD partitioning;
3. continual_metrics — R-matrix, BWT/FWT/AIA, paired/hierarchical bootstrap, power/SESOI, resource/Pareto reporting;
4. diagnostics — trace schema, representation drift/geometry, behavioral fingerprints, causal state-patching hooks.

002C then consumes those diagnostics for mechanism-level causal tests instead of implementing one-off probes in each architecture.

Claim gate upgrade

An architecture-level claim should ideally satisfy all of:

• powered in advance for a preregistered SESOI;
• replicated across independent environment programs;
• survives OOD_COMPOSITION and at least one OOD_FAMILY axis;
• survives strong/simple baselines at matched resource budget;
• component ablation predicts the loss;
• internal causal intervention restores/degrades the capability in the predicted direction;
• simpler alternatives do not sit on the same Pareto front at materially lower complexity;
• confirmatory and replication streams remain untouched until their designated runs.

If we can meet that bar, the result becomes much more than Symthaea scored higher: it becomes evidence that a particular internal mechanism causally supports a specific generalizable capability under conditions predicted by the architecture.

[Tristan-Stoltz-ERC]

Further refinement — placebo mechanisms, claim ledger, and differential validation

This pass should make it harder for a positive result to survive unless the proposed mechanism is genuinely doing the work.

1. Add placebo/sham mechanisms

A normal ablation asks whether removing a mechanism hurts. Add matched placebo interventions that preserve parameter/state/compute footprint while destroying the mechanism's semantics.

Examples:

• tau_sham: preserve the same distribution/range of tau updates, but permute their assignment across semantic groups or time steps;
• hebbian_sham: execute equal-magnitude local updates using permuted/unrelated pre-post pairs;
• memory_sham: perform the same number of retrieval operations but return a matched-distribution irrelevant memory;
• context_sham: inject a context vector with the same norm/statistics but unrelated to the active regime;
• patch_sham: transplant a state from an irrelevant matched example/component rather than the causally corresponding clean state;
• hdc_role_sham: preserve the same HDC operations but randomly permute role bindings so algebraic capacity/compute stays similar while intended relational semantics are broken.

Interpretation rule: if a sham intervention preserves the same behavioral gain as the real mechanism, that mechanism is not causally identified. Return a qualification such as PLACEBO_EXPLAINS_GAIN or MECHANISM_NOT_IDENTIFIED rather than granting a mechanism-level claim.

This complements ordinary off/on ablations because it distinguishes 'the component matters' from 'the claimed computation performed by the component matters.'

2. Reuse the Butlin evidence philosophy as a generic claim ledger

BUTLIN_EVIDENCE_TIER_DESIGN.md already contains a strong pattern: separate architectural existence, observation, causal support, functional support, contradiction, and inconclusive measurement; never collapse unlike evidence into one scalar.

Generalize that philosophy rather than copying Butlin types literally.

Suggested SYM-ARCH evidence model:

Claim
  id
  statement
  target_mechanism
  preregistered_primary_metric
  primary_comparator
  SESOI
  required_validity_gates[]
  required_interventions[]
  disqualifiers[]
  evidence_refs[]
  outcome
  annotations[]

Keep orthogonal evidence dimensions separate:

• empirical outcome: SUPPORTED | NOT_DEMONSTRATED | CONTRADICTED | INCONCLUSIVE;
• support kind: OBSERVATIONAL | ABLATION_CAUSAL | INTERVENTION_CAUSAL | FUNCTIONAL;
• generalization status: DEV_ONLY | IID_CONFIRM | OOD_CONFIRM | REPLICATED;
• resource status: WITHIN_BUDGET | DOMINATED | UNKNOWN;
• provenance status: VALID | OBSERVED_HOLDOUT | MALFORMED | UNKNOWN.

Do not invent a weighted aggregate across these axes. A claim should be reportable as, for example:

SUPPORTED / INTERVENTION_CAUSAL / OOD_CONFIRM / WITHIN_BUDGET / VALID

rather than '0.87 evidence score.'

Every published/plain-language claim should be generated from this ledger or point back to a ledger entry and immutable evidence artifact.

3. Add two-way causal intervention tests

For component-state patching, require both directions where meaningful:

• necessity/degradation: insert the corrupted/placebo component into an otherwise clean execution and show the target behavior degrades;
• sufficiency/recovery: insert the clean/counterfactual component into the corrupted execution and show target behavior recovers.

Also require specificity:

• patching the target component should affect the predicted behavior;
• patching irrelevant matched components should minimally affect it;
• the intervention should not globally improve/degrade unrelated competencies.

Record a component-level causal effect matrix rather than one patch score:

                 target behavior   unrelated behavior A   unrelated behavior B
HDC relation          large                small                  small
short-tau state       task-specific        small                  small
long-tau state        retention            small                  small
memory retrieval      delayed recall       small                  small

The desired evidence is selective causal influence, not merely sensitivity to perturbation.

4. Differential implementations for scientific-critical code

For the pieces capable of manufacturing a scientific result, use two independent implementations where feasible:

• R[t_train][t_eval] metric computation;
• BWT/FWT/AIA derivation;
• paired delta aggregation;
• hierarchical bootstrap / CI calculation;
• task-program oracle;
• task split / leakage detector;
• environment manifest hashing;
• result classifier / claim-gate evaluation.

One can be a simple slow/reference implementation and the other the production implementation. Run identical synthetic fixtures and randomized property tests through both.

If they disagree beyond exact/tolerance contracts, the run is INCONCLUSIVE_METRIC_IMPLEMENTATION_MISMATCH and no claim is emitted.

This protects against a subtle but serious failure mode: a bug in the evaluator becoming an apparent architecture discovery.

5. Benchmark mutation tests

Test the benchmark validator itself by deliberately injecting known defects and requiring it to catch them:

• train/test leakage;
• constant-label task;
• accidental task-ID feature;
• duplicated-equivalent program;
• broken oracle;
• shuffled labels;
• timestamp leakage of upcoming boundary;
• impossible intervention;
• malformed provenance/manifest;
• hidden state reset accidentally enabled/disabled;
• one baseline receiving extra observations or replay bytes.

The validation suite only earns trust if known benchmark pathologies make it fail closed.

6. Robustness/specification envelope on DEV only

Use DEV to sweep non-theoretical nuisance choices across reasonable prespecified ranges:

• readout/update rate;
• threshold/window definitions;
• presentation order;
• HDC dimension;
• state width;
• timing jitter magnitude;
• modest noise levels.

Report an effect-direction/effect-size distribution rather than selecting the best configuration.

Before CONFIRM, freeze:

• one canonical operating point;
• the nuisance-range sensitivity checks that will be run;
• which checks are confirmatory vs diagnostic.

A mechanism-level claim should be downgraded if it only appears in a narrow knife-edge configuration unless the theory specifically predicts that regime.

7. Prespecify peeking and stopping behavior

Do not repeatedly inspect partial confirmatory results and stop when they become favorable.

Use either:

• a fixed confirmatory environment count determined by the DEV power study; or
• a formally defined always-valid/sequential stopping procedure if compute savings justify the extra complexity.

Infrastructure-only reruns remain allowed only before behavioral outputs are exposed, consistent with the existing DEV/CONFIRM/REPL policy.

8. Architecture-claim gate v2

A strong mechanism claim should require all of:

1. benchmark validity gates pass;
2. confirmatory environment count was prospectively powered;
3. paired/hierarchical CI clears the preregistered SESOI;
4. ordinary ablation shows necessity;
5. placebo/sham ablation rules out generic component/compute effects;
6. state intervention/patching provides selective causal evidence where applicable;
7. the result survives generator-level OOD;
8. the effect is not dominated by a simpler baseline under the matched resource budget;
9. metric/oracle differential implementations agree;
10. independent REPL reproduces the direction and practically meaningful magnitude.

If any of 1/5/9 fail, classify INCONCLUSIVE rather than negative. If 3/4/7/10 fail with valid measurement, classify the narrower null/negative result honestly.

9. Keep scope under control

These are mostly 002A/002C infrastructure additions; they should not create more top-level tranches. Recommended implementation sequence now:

1. evidence/claim schema + provenance registry;
2. TaskProgram + oracle/validator + mutation tests;
3. continual metrics + paired/hierarchical stats + prospective power;
4. differential/reference evaluators;
5. diagnostic trace + intervention API;
6. placebo mechanism API;
7. only then expand baseline and architecture experiments.

This gives 002 a strong property: every new mechanism can be tested under the same anti-shortcut, anti-leakage, anti-metric-bug, anti-placebo framework.

[Tristan-Stoltz-ERC]

Merge discipline for SYM-ARCH research

Add an explicit merge policy so research infrastructure and empirical claims do not blur together.

Infrastructure PRs

May merge when:

• implementation correctness gates pass;
• deterministic fixtures/reference implementation checks pass;
• no confirmatory behavioral metrics are required for correctness;
• provenance/evidence schema is stable;
• the PR does not silently change a frozen experiment's hypothesis, thresholds, task generator, or analysis.

Infrastructure correctness is not itself evidence for an architectural capability.

Confirmatory experiment PRs

May merge only after:

• the exact-head preregistered workflow has completed;
• its evidence artifact/schema is valid and hash-bound;
• the result is frozen as produced, including NULL/NEGATIVE/INCONCLUSIVE outcomes;
• no post-result threshold/config edits occurred under the same experiment version;
• any rerun status is classified (infrastructure-only retry, post-hoc, etc.).

A negative result is mergeable evidence; a missing/failed experiment run is not.

Claim-bearing follow-up PRs

May merge strong claim wording only if the relevant ClaimLedger entry has the required support state (construct-validity gates, prospective power, strong baseline, causal/mechanism tests as applicable, OOD/replication/resource gates). Otherwise merge the implementation/evidence with narrower wording.

Current application

PR #53 / SYM-ARCH-001 should remain unmerged while its dedicated exact-head workflow is queued. Once the preregistered run completes successfully and emits valid evidence, the result should be merged regardless of whether the mechanical verdict is PASS, MIXED, NULL, or NEGATIVE. The scientific value is in preserving the frozen result, not requiring a favorable one.

Do not auto-merge #53 unless the dedicated SYM-ARCH workflow itself is a required merge condition; otherwise repository policy could merge the PR before the experiment actually executes.

[Tristan-Stoltz-ERC]

Implementation started — SYM-ARCH-002A core

Opened PR #57 (research: add SYM-ARCH-002A experimental core) from current main.

Scope is intentionally infrastructure-only and independent of SYM-ARCH-001 / PR #53:

• DEV / CONFIRM / REPL experiment manifests and first-use hygiene;
• separate environment / representation / learner / stream seed namespaces;
• typed, hashable TaskProgram identity contract;
• full T+1 x T continual-learning performance matrix with AIA/BWT/FWT/forgetting;
• paired candidate-minus-control BCa deltas reusing existing psych-bench bootstrap code;
• non-scalar ClaimLedger with evidence/generalization/replication/resource/provenance dimensions and claim-wording ceilings;
• focused exact-head CI and design/acceptance document.

PR #57 carries no architecture winner/loser claim. It should merge on infrastructure correctness once its exact-head focused workflow passes.

PR #53 remains frozen and unmerged while its preregistered SYM-ARCH-001 workflow is still queued. Do not alter its scientific configuration or merge it before the evidence run completes.

[Tristan-Stoltz-ERC]

Implementation update — SYM-ARCH-002A core hardened and scope-frozen

PR #57 now implements the first experimental-core tranche and has been hardened beyond the initial scaffold:

• confirmatory paired inference now has a strict environment-level API (paired_environment_delta_bca) that requires one aggregate observation per unique generated environment and rejects duplicate environment identities, preventing nested RNG runs from becoming pseudoreplicates;
• SESOI interpretation is interval-based: practical gain/regression requires the complete CI to clear the preregistered margin, while Equivalent requires the complete CI to lie inside the equivalence band; v1 explicitly does not claim formal TOST;
• task support validation rejects empty/duplicate descriptors and can fail closed on accidental train/evaluation overlap;
• ClaimEvidenceBinding ties ClaimLedger entries to the exact experiment-manifest digest, task-program digests, result-artifact digest, and analysis code revision, rejecting mismatched claim ids and duplicate/malformed evidence identities;
• the dedicated workflow formats and tests the confirmatory helper path in addition to the base experiment module.

The conceptual scope of #57 is now frozen. Hierarchical nested-run bootstrap, prospective power, mutation-tested procedural worlds, Pareto/resource analysis, diagnostics/state interventions, and stronger external baselines should land as later focused tranches instead of growing the foundation PR further.

Current exact head: a84dc8c384fac5c1d3e0fa4240ff648eb39f5aee.

The repository Actions queue is still preventing execution; the exact-head 002A workflow has been triggered but remains queued. No CI success is being inferred from queue status.

[Tristan-Stoltz-ERC]

Implementation update — SYM-ARCH-002A2 hierarchical statistics + prospective power

Opened draft stacked PR #58 on top of #57.

A2 adds:

• generated environments as the explicit independent unit;
• paired nuisance-run identities inside each environment;
• equal-environment-weight hierarchical bootstrap with nested run resampling;
• explicit hierarchical percentile interval naming (no silent BCa relabeling);
• DEV-only prospective power planning;
• manifest/result environment-count binding;
• separately frozen planning_effect beyond the SESOI so the observed DEV mean cannot become a winner's-curse alternative hypothesis;
• residual_scale >= 1.0 for conservative DEV-variance sensitivity;
• Monte Carlo power Wilson intervals;
• conservative sustained threshold selection: the lower power bound must clear target at the chosen environment count and every larger tested count;
• DEV manifest + complete planning-input digests for later CONFIRM provenance binding.

No actual CONFIRM sample size is selected by this PR because no qualifying DEV campaign is part of it. The planning effect must be justified/frozen outside the code path.

PR #58 is intentionally draft and stacked on #57. After #57 merges it should be rebased/retargeted to main without changing the statistical contract.

Exact initial A2 head: f949c00d1eb06a9c14fac73a5ddea4ad77205b50.

Dedicated SYM-ARCH-002A2 Hierarchical Power workflow has triggered but is currently queued; no CI success is inferred from queue status.

Next focused 002A work after A2 should be procedural benchmark validity/oracle + mutation testing, not more statistics in #58.

[Tristan-Stoltz-ERC]

Implementation update — SYM-ARCH-002A3 benchmark validity + mutation testing

Opened draft sibling-stack PR #59 directly on #57, independent of #58's statistics work.

A3 adds:

• executable v1 RuleExpr symbolic oracle;
• oracle digest bound to the exact executable rule;
• deterministic generated-dataset and feature-only identities;
• explicit task-free leakage policy;
• fail-closed checks for program/oracle provenance, empty/invalid examples, duplicate ids, train/eval feature leakage, forbidden task/world/boundary features, undeclared support, oracle-label mismatches, class-count mismatch, and split class degeneracy;
• five deliberate benchmark corruptions and a mutation-detection suite that requires the unmodified benchmark to validate before testing failure paths.

The wording ceiling is intentionally narrow: passing A3 v1 means structural/oracle benchmark integrity passed, not that every statistical shortcut has been ruled out. Marginal predictors, nearest-neighbor/lookup, shuffled-relation controls, semantic-equivalence checks, and temporal leakage probes remain later construct-validity work.

PR #59 is draft and stacked on #57; after #57 merges it should be rebased/retargeted to main without changing the validity contract. Its dedicated workflow has triggered and is currently queued.

Separately, SYM-ARCH-001 run #2 finally executed but failed before campaign execution on an unrelated stale LoopTrialResult.cycle_reward synthetic test fixture. No behavioral metrics/artifact were produced. PR #53 now has a one-commit infrastructure-only repair (eb931fbaed960f066406caf969c5aaad350c9d64), machine-verified as one file / 5 deletions relative to the frozen scientific head, and run #3 is queued. Scientific settings remain unchanged.

[Tristan-Stoltz-ERC]

SYM-ARCH-002A4 implemented — adversarial construct validity

Opened draft stacked PR #60: research: add SYM-ARCH-002A4 adversarial construct-validity controls.

Stack: #57 (002A core) → #59 (A3 structural/oracle validity) → #60 (A4 construct validity). #58 remains the sibling hierarchical-statistics stack on #57.

A4 scientific contract

A4 makes generated benchmarks fail closed as INCONCLUSIVE_BENCHMARK before any architecture result is interpreted when a trivial train-only explanation remains viable.

Implemented controls:

• deterministic fair-chance sanity control;
• training-majority predictor;
• single-feature marginal predictor, with feature selection using training performance only;
• exact feature-assignment lookup;
• deterministic categorical nearest neighbor;
• prevalence-preserving shuffled-label single-feature controls;
• prevalence-preserving shuffled-label nearest-neighbor controls;
• executable symbolic-oracle positive control.

Fitted shortcut models consume learner-visible features only. example_id, support_tags, support metadata, oracle outputs, and evaluation labels are excluded from fitted inputs/model selection.

Finite-sample resolution

A4 computes the 95% Wilson upper accuracy bound for a fair Bernoulli classifier at the actual evaluation-set size. If ordinary chance uncertainty reaches the tighter preregistered shortcut ceiling, the benchmark is automatically inconclusive and must increase evaluation support.

This intentionally prevents very small held-outs (including four-item designs) from supporting fine-grained shortcut/architecture claims.

Threshold hygiene

Claim-bearing use must freeze before result inspection:

• shortcut accuracy ceiling;
• shuffled-label mean ceiling;
• oracle floor;
• chance seed;
• complete unique shuffle-seed set;
• A3 structural policy;
• generator/version and seed manifest.

The 0.80 thresholds in unit tests are synthetic fixtures only, not scientific defaults.

Tests

Tests cover a clean XOR-style benchmark, injected single-feature label channel, insufficient four-item evaluation resolution, support-metadata isolation, prevalence-preserving shuffles, and invalid shuffle-policy contracts.

Cross-stack compile repair

The first real #53 hosted run exposed an unrelated stale LoopTrialResult::cycle_reward field in harness/neuromod_correlation.rs; the preregistered campaign never started and no behavioral metrics were observed.

The exact same fixed file blob (c6056aad3d20d0ad43cbd1a38fddbaf3267cd173) has now been propagated to #53, #57, #58, #59, and #60. No architecture, statistics, validity, threshold, seed, or decision-rule logic was changed by that repair.

Stack ancestry was then normalized with no-content merge commits so #58/#59/#60 diffs contain only their intended tranches.

Current exact heads / dedicated runs:

• research: run SYM-ARCH-001 continual compositional adaptation #53 SYM-ARCH-001: eb931fbaed960f066406caf969c5aaad350c9d64, run 32825418156 — queued;
• research: add SYM-ARCH-002A experimental core #57 002A core: 8b80585e0babf6f62ace79f79dd0203e2853b38e, run 32829075229 — queued;
• research: add SYM-ARCH-002A2 hierarchical statistics and power planning #58 A2 statistics: 42a55674582d18924b1bdeccde3538eff931e750, run 32829475115 — queued;
• research: add SYM-ARCH-002A3 benchmark validity and mutation testing #59 A3 validity: 4dc97de69e873254f879c60ba3b8a700cae9cd28, run 32829492270 — queued;
• research: add SYM-ARCH-002A4 adversarial construct-validity controls #60 A4 construct validity: 3de0ad5e515981e385f7fa83f1818b7dc88ee187, run 32829511391 — queued.

No CI success or architecture result is being inferred from queued status.

Next boundary

Do not expand A4 further while its exact-head gate is pending. The next high-value tranche after instrumentation validation should be a differential/reference evaluator or the first strong-simple baseline tranche (fixed random features + analytic/RLS readout), depending on which dependency is cleaner after #57/#59/#60 execute.