Skip to content

research: add SYM-ARCH-002A6 online measurement dynamics - #65

Draft
Tristan-Stoltz-ERC wants to merge 11 commits into
research/sym-arch-002a-core-v1from
research/sym-arch-002a6-measurement-dynamics-v1
Draft

research: add SYM-ARCH-002A6 online measurement dynamics#65
Tristan-Stoltz-ERC wants to merge 11 commits into
research/sym-arch-002a-core-v1from
research/sym-arch-002a6-measurement-dynamics-v1

Conversation

@Tristan-Stoltz-ERC

@Tristan-Stoltz-ERC Tristan-Stoltz-ERC commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds SYM-ARCH-002A6 online measurement dynamics as a draft sibling tranche directly on the frozen 002A core (#57).

This PR adds no architecture mechanism and consumes no CONFIRM data. It closes measurement gaps exposed by the negative SYM-ARCH-001 result, especially the ambiguity between low forgetting and weak acquisition.

Prequential acquisition contract

Every online-learning step is measured in this order:

  1. predict before consuming the current label;
  2. record correctness and inference latency;
  3. apply the labeled update;
  4. record update latency;
  5. record post-update resource state.

Post-update correctness on the same item is not admissible as acquisition evidence.

Provenance-bound phase traces

Each OnlineMeasurementTrace binds:

  • experiment-manifest digest;
  • task-program digest;
  • runtime-context digest;
  • contiguous phase identity;
  • optional initial learner/spec-state digest;
  • ordered prequential steps.

The trace receives its own domain-separated BLAKE3 digest.

Acquisition metrics

A6 adds:

  • sustained examples-to-criterion using a frozen rolling window, accuracy threshold, and required consecutive qualifying windows;
  • overall prequential accuracy;
  • terminal-window accuracy;
  • normalized cumulative-accuracy AUC that rewards earlier acquisition at matched overall accuracy.

The cumulative-AUC definition is frozen and cannot be silently replaced by ordinary mean correctness or post-update accuracy.

Criterion validity

experiment_measurement_validity audits the frozen acquisition criterion against a preregistered simple-reference accuracy ceiling and a minimum practical excess.

It exposes:

  • integer correct count actually required by the finite window;
  • effective finite-window accuracy threshold;
  • effective excess above the reference ceiling;
  • admissible vs reference_confounded status with qualifiers.

A claim-bearing examples-to-criterion result may not use a reference_confounded criterion.

This is a construct-validity guard only; A4 shortcut controls and A2 uncertainty/power remain required.

Latency / throughput

For inference and update latency independently:

  • sample count;
  • total nanoseconds;
  • mean;
  • p50;
  • p95;
  • observations/second when measured time is nonzero.

Latency traces are runtime-context-bound. A6 does not authorize cross-hardware latency claims from unmatched runtime-context digests.

Resource trace

Post-update snapshots record:

  • trainable scalar parameters;
  • persistent state bytes;
  • replay bytes;
  • temporal state bytes;
  • optional RSS.

Summaries report final/peak trainable and persistent state plus peak replay, temporal state, and observed RSS.

Scientific boundary

A6 complements rather than replaces the continual R[t_train][t_eval] matrix:

  • R matrix → retention/transfer across tasks;
  • A6 → acquisition/recovery dynamics inside each phase.

A system with low forgetting but slow/absent criterion attainment can therefore no longer be described as strong continual learning merely because retention-style metrics look stable.

CI

Dedicated exact-head workflow gates:

  • cargo fmt --all -- --check;
  • focused experiment_measurement tests;
  • focused experiment_measurement_validity tests;
  • full symthaea-psych-bench library check on Rust 1.96.

Wording ceiling

Merging this PR supports only:

Symthaea psych-bench contains provenance-bound prequential acquisition, criterion-validity, latency/throughput, and resource measurement primitives suitable for later preregistered continual-learning experiments.

It does not support a Symthaea performance claim, a faster-learning claim, a hardware-independent latency claim, a resource-efficiency claim, or a claim-bearing threshold that fails the reference audit.

Tracks #55. Depends only on #57.

Integration-only parent sync; no A6 scientific changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant