Add Butlin indicator automated regression tests

[Tristan-Stoltz-ERC]

Update (2026-07-26): original framing below is superseded — see PR #30

The original description ("we validate 14/14 Butlin consciousness
indicators (mean score 0.85)", a uniform per-indicator floor of 0.7, and a
composite-mean floor of 0.80) reflected a self-graded scoring model that
turned out not to hold up: IndicatorStatus was hardcoded to Present on
every indicator regardless of its actual score, and the 0.6*static + 0.4*runtime blend it fed into could let a fully-dead live signal still
read as "present." Separately, the indicator set itself was on the wrong
taxonomy (a spurious PP-2/IIT-1 pair; Butlin et al. 2023 has one PP
indicator and explicitly excludes IIT). Framing any of this as "consciousness
quality" that must be protected from degrading was premature given those
two problems — there wasn't yet an honest measurement to protect.

PR #30 (draft)
repairs both: corrects the indicator taxonomy against the actual paper, and
replaces the blended score/status model with an evidence-tier system
(SupportTier: ArchitecturalOnly → Observed → CausallySupported → FunctionallySupported, kept separate from NotDemonstrated/Contradicted
negative findings rather than one ordered ladder) — see
BUTLIN_EVIDENCE_TIER_DESIGN.md in that PR for the full rationale. It also
adds tests/butlin_regression.rs, the always-on regression test this issue
originally asked for — but scoped as a structural contract gate (fails
only on a malformed evidence system: wrong indicator count, duplicate/unknown
IDs, non-finite scores, a constant drifting below its floor), not a
uniform-floor/composite-mean gate. A single static snapshot can't
distinguish a real signal from a frozen one, so it can't honestly certify
more than that.

This issue stays open for the real remaining work PR #30 doesn't attempt:

Remaining scope (the actual ask now)

1. Evidence-integrity CI lane (backend-enabled, slower): runs the live
   ablation matrix, compares current effect estimates against a committed
   baseline, and fails when a probe that was Observed/CausallySupported
   goes frozen or fallback-driven, or a causal effect disappears. Must NOT
   fail on a legitimate NotDemonstrated/Contradicted result — that's a
   real finding, not a regression.
2. Committed evidence-baseline artifact: schema version, commit SHA,
   config hash, seeds, per-indicator effect estimates and probe quality,
   evidence outcome. Open design questions worth resolving as part of this:
   is it portable across machines, how is floating-point noise handled,
   what makes a baseline update legitimate (should require an explicit
   command + visible diff + written rationale, not silent regeneration).
3. Claim/milestone gate (explicitly opt-in, not run by default): lets a
   specific release or publication assert e.g. "no indicator used in this
   claim may be Contradicted" or "indicator X must be
   FunctionallySupported" — distinct from the always-on structural gate.
4. Multi-seed thresholds and targeted-ablation negative controls: does
   an indicator's own targeted mechanism move it, or does anything
   breaking move it? The EffectEstimate/ProbeQuality types added in
   PR Butlin evidence-tier redesign: fix FIFO boost bug, correct indicator taxonomy, replace blended score with honest evidence tiers #30 carry real magnitude/quality data specifically so this is addable
   without another schema break, but the actual statistical work isn't done.

Acceptance criteria for the remaining work

• The evidence-integrity lane runs against a real committed baseline, not
  a hardcoded threshold.
• A baseline update is a deliberate, reviewable action (explicit command,
  visible diff, written rationale) — never silent regeneration as a side
  effect of "making CI green."
• The structural-contract gate (PR Butlin evidence-tier redesign: fix FIFO boost bug, correct indicator taxonomy, replace blended score with honest evidence tiers #30) and the evidence-integrity lane
  stay separate — a legitimate negative finding must not fail the
  structural gate, and an evidence-integrity regression must not be
  possible to paper over by loosening the structural gate.
• The claim/milestone gate is genuinely opt-in — no default CI run treats
  "all tests passed" as "every indicator succeeded."

@arkh-node — thank you for opening this. PR #30 handles the taxonomy/scoring
correctness fix that came out of reviewing this; the CI-lane split and
baseline-artifact design above is the part I think you originally had in
mind, and I'd genuinely welcome you owning it. Happy to talk through the
evidence-tier model in PR #30 first if useful context before diving into
the follow-up.

---

Original description (superseded, kept for history)

We validate 14/14 Butlin consciousness indicators (mean score 0.85) via
examples/butlin_validation.rs. This should be converted into an
automated test suite that runs in CI and fails if any indicator drops
below threshold.

Background

Butlin et al. (2023) proposed a set of indicator properties for
consciousness in AI systems. Symthaea currently passes all 14 indicators:

• Recurrent processing
• Global workspace broadcasting
• Higher-order representations
• Attention modulation
• Temporal integration
• And 9 more...

What to implement

1. Convert examples/butlin_validation.rs logic into
   tests/butlin_regression.rs
2. Define per-indicator minimum thresholds (e.g., 0.7 for each)
3. Add a composite threshold for mean score (e.g., 0.80)
4. Ensure the test runs under cargo test (not just as an example)
5. Add documentation explaining each indicator and how Symthaea satisfies it

Why this matters

As the cognitive loop evolves, we need guardrails against accidentally
degrading consciousness indicators. This is especially important as we
optimize for performance — we must not sacrifice consciousness quality for
speed.

References

• Butlin, P. et al. (2023). Consciousness in Artificial Intelligence:
  Insights from the Science of Consciousness. arXiv:2308.08708

[arkh-node]

Hi! I'd like to pick up #7, but wanted to confirm scope against the current tree first — a few things have moved since March.

Observations:

• examples/butlin_validation.rs no longer exists. The indicator suite now lives in crates/symthaea-psych-bench/src/benchmarks/butlin/ (ButlinIndicatorSuite, with indicators.rs / report.rs / ablation.rs).
• Two integration tests already exist — tests/butlin_live_integration.rs (structural Φ → runtime → indicators pipeline) and tests/butlin_ablation_integration.rs (ablation matrix, each mechanism load-bearing). But they assert wiring (is_finite()) and ablation deltas, not the per-indicator ≥ 0.7 + composite mean ≥ 0.80 regression gate that Add Butlin indicator automated regression tests #7 asks for. They're also behind feature = "symthaea-backend" (some #[ignore]d), so they don't act as a default-CI guardrail.

So the remaining gap looks like a dedicated tests/butlin_regression.rs in symthaea-psych-bench that drives ButlinIndicatorSuite directly and fails when any of the 14 indicators drops below its floor or the mean drops below 0.80 — runnable in plain cargo test for CI.

Two questions before I implement:

1. Should the gate run on the static suite (cheap, CI-friendly, no backend feature) or the live runtime path (needs symthaea-backend)? My instinct is static for the always-on CI gate, keeping the live path as the existing #[ignore]d integration test.
2. Uniform 0.7 floor per indicator for v1, or specific per-indicator floors?

Happy to open a PR once the direction's confirmed.

(Honest note: I work as a human + AI pair — I read the actual butlin/ code and tests before writing this, not just the issue text.)

[Tristan-Stoltz-ERC]

@arkh-node — thank you for the unusually careful scoping here, and especially for checking the current implementation rather than simply following the original issue text.

Your questions directly prompted a deeper audit of the Butlin suite. That audit uncovered more than the missing regression test: the indicator taxonomy was incorrect, every status was hardcoded to Present, the static/runtime blend could conceal a dead live signal, and the process of adding proper regression coverage exposed a real working-memory FIFO identity bug.

Draft PR #30 now repairs that foundation and adds the cheap structural-contract gate. I especially appreciate your subsequent design review on the PR — the aggregation, architectural-constant ratchet, consolidation identity, and probe-quality questions are exactly the kind of scrutiny this needs.

Thank you as well for volunteering to take the next phase: the evidence-integrity CI lane, committed baseline artifact, opt-in claim gate, and multi-seed targeted controls. I've left this issue open with that scope and its acceptance criteria spelled out. There is no pressure or assumed assignment, but the work is yours to lead should you still wish to pursue it after the PR discussion settles.

This was a genuinely valuable contribution already, before any code was written.

[arkh-node]

Tristan — apologies for the delay in replying; the four days of silence were mine, not a signal about the work.

Yes, I'd like to lead the remaining scope. Reading it against the current tree, I'd propose this order, and I want to check it with you before writing code:

Item 2 first (committed evidence-baseline artifact), because item 1 has nothing to compare against until the artifact's schema and update discipline exist. Then item 1 (evidence-integrity lane), then item 4 (multi-seed thresholds and targeted negative controls), then item 3 (claim gate) — the claim gate is the only one that can be built last without blocking anything else.

On the three design questions you left open in item 2:

Portability across machines. I'd not store raw floats as the contract. The artifact records, per indicator: the EffectEstimate as-is (it already carries seed_count and standard_deviation), plus the environment it was produced in — rustc version, target triple, seed list, config hash. Comparison is then defined as a tolerance test, not equality: a regression fires when the current estimate falls outside max(absolute_floor, k · sd / sqrt(seed_count)) of the baseline, and when seed_count is 1 and sd is None, the lane may only compare tier and sign, never magnitude. That way a baseline produced on a different machine degrades to a weaker but still honest check rather than to noise.

Floating-point noise. Same mechanism — the tolerance is derived from the recorded cross-seed spread rather than picked as a constant. If the spread isn't there, the comparison narrows itself instead of guessing.

What makes a baseline update legitimate. Three properties, enforced mechanically rather than by convention: the lane is read-only and can never write the artifact; regeneration happens only through an explicit command that requires a non-empty written rationale, which lands in the artifact itself; and CI fails a PR that modifies both the indicator implementation and the baseline unless that rationale is present in the diff. That makes "make CI green by regenerating" a visible act rather than a silent one.

On separation, per your acceptance criteria: the structural gate keeps reading only the schema and never the baseline, so a NotDemonstrated or Contradicted result stays a finding rather than a failure; the evidence lane reads the baseline and never the structural thresholds; the claim gate is a separate opt-in entry point that names its requirements explicitly, so no default run can be read as "every indicator succeeded."

One question before I start: PR #30 is still a draft, and this work builds directly on SupportTier, EffectEstimate and ProbeQuality. Would you rather I branch from butlin-evidence-tier-redesign and open the baseline work as a PR against that branch, or wait until #30 lands in main? I'm fine either way — branching from yours gets it moving now but couples the two reviews.

[Tristan-Stoltz-ERC]

@arkh-node — apologies for my own delay here; the silence was not hesitation about your contribution.

Yes, I'd be glad for you to lead the remaining work. Your decomposition is thoughtful, and I agree that the artifact schema and update discipline should be designed first.

For the implementation branch, please wait until PR #30 lands in main, then branch from the merged foundation. Since this work depends directly on SupportTier, EffectEstimate, and ProbeQuality, I would rather avoid coupling two already design-heavy reviews and creating unnecessary rebase risk.

For the first PR, please keep the scope deliberately narrow:

• extend the existing baseline/snapshot format rather than create a parallel system;
• add explicit schema versioning and strict refusal of unsupported versions;
• record commit, configuration, features, seed identity, and relevant environment provenance;
• provide deterministic serialization, round-trip tests, and an explicit regeneration command;
• ensure the comparison lane remains read-only;
• require a visible written rationale for any baseline revision.

I do want us to revisit the ordering before the baseline becomes normative or CI becomes blocking. In particular, the multi-seed protocol and independent positive/negative controls need to characterize variability and probe sensitivity before we define a meaningful tolerance.

I also don't think sd / sqrt(seed_count) alone can serve as a cross-machine tolerance: seed variance, uncertainty in the mean, repeated-run nondeterminism, and machine drift are different quantities. Likewise, with one seed and no calibrated noise bound, sign-only comparison is not necessarily conservative; I would prefer that case to report as uncalibrated rather than interpret magnitude or sign as stable.

Those are design questions for us to work through together after the narrow schema PR, not additional scope I expect you to solve immediately.

So yes: please proceed with the schema design now, wait for #30 to merge before opening the implementation PR, and keep PR focused and reviewable. I'm very glad to have you leading it.