Skip to content

RFC: CLI-managed regression baselines (committed + native bundles)#5

Open
lewisjared wants to merge 8 commits into
mainfrom
rfc/regression-baselines
Open

RFC: CLI-managed regression baselines (committed + native bundles)#5
lewisjared wants to merge 8 commits into
mainfrom
rfc/regression-baselines

Conversation

@lewisjared

@lewisjared lewisjared commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Adds an RFC for a more structured approach to tracking regression baselines per test-case.

The output of each test case consists of two bundels:

  • Committed bundle (series.json + CMEC bundles) - tracked in git and part of the PR diff.
  • Native bundle (raw outputs) - minted via the CI and stored as publicaly accessible results.

Key decisions

  • test_case_version (int in manifest.json) — an explicit bump is the re-baseline trigger; CI hard-fails a changed bundle with no bump.
  • Trying to avoid the use of a shared-runner on PRs to allow PRs from forks. The fallback is that the nightly check will run on a shared-runner so should be able the catch any drifts for diagnostics that can't run on public runners.
  • NativeStore Protocol — public read / gated write; backend deferred (R2 to be used for a prototype).
  • Consumes execution.json from #3 for baseline provenance once it lands.

@lewisjared lewisjared changed the title RFC: CLI-managed regression baselines (golden-in-git + native fixtures) RFC: CLI-managed regression baselines (committed + native bundles) Jun 4, 2026
@lewisjared

Copy link
Copy Markdown
Contributor Author

@mikapfl @fuchsi-huber Interested in your thoughts on this too following our discussions about regression outputs.

@bouweandela The regression outputs need some more rework to be fit for purpose. The gist of the changes for a diagnostic dev is that we track a smaller set of the diagnostic output (no logs, no images) to avoid bloating the git repo. We can do this now.

The rest of the workflow can come later and is more helpful from the view of a reviewer/maintainer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant