[EP-01] S-13: Golden Fixtures and Regression Suite

[aryeko]

S-13: Golden Fixtures and Regression Suite

Last updated: 2025-09-21

Outcome

• Lock deterministic sample runs (tiny and medium universes) into golden fixtures, enabling regression detection across the pipeline.

Deliverables

• Fixture datasets (raw inputs, configs, holdings) checked into tests/fixtures/ with documented provenance.
• Golden outputs for curated data, signals, risk alerts, rebalance proposals, reports, manifests, and notifications.
• Automation (poetry run ts fixtures verify) that replays the pipeline on fixtures and compares generated artifacts against goldens with clear diff tooling.

Functional Requirements

• Provide tooling to update goldens intentionally (ts fixtures update --scenario tiny) with reviewable diffs.
• Comparisons must ignore known volatile fields (timestamps) or normalize them before diffing.
• Integrate regression suite into CI so unexpected diffs fail the build with contextual logging and artifact uploads.
• Document process for adding new scenarios and regenerating artifacts consistently.

CLI Additions

• poetry run ts fixtures verify --scenario tiny runs the pipeline against the tiny fixture set and validates outputs against stored goldens.
• poetry run ts fixtures update --scenario medium regenerates goldens after intentional changes, prompting contributors to review diffs.

Verification

1. Run ts fixtures verify for each scenario; confirm CI fails when outputs drift.
2. Intentionally tweak a golden (e.g., change price) and observe verify command failing with a diff summary.
3. Use the update command to regenerate and commit goldens; ensure manifest/log updates remain deterministic.
4. Add unit tests for fixture loading utilities and diff reporting helpers.

Dependencies

• S-03 through S-12 (pipeline completeness required).

Notes

• Keep fixture universes minimal to maintain fast CI while still exercising key code paths (multi-symbol, benchmark, rebalance day, no-rebalance day).