RFC: CLI-managed regression baselines (committed + native bundles)#5
Open
lewisjared wants to merge 8 commits into
Open
RFC: CLI-managed regression baselines (committed + native bundles)#5lewisjared wants to merge 8 commits into
lewisjared wants to merge 8 commits into
Conversation
Contributor
Author
|
@mikapfl @fuchsi-huber Interested in your thoughts on this too following our discussions about regression outputs. @bouweandela The regression outputs need some more rework to be fit for purpose. The gist of the changes for a diagnostic dev is that we track a smaller set of the diagnostic output (no logs, no images) to avoid bloating the git repo. We can do this now. The rest of the workflow can come later and is more helpful from the view of a reviewer/maintainer. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an RFC for a more structured approach to tracking regression baselines per test-case.
The output of each test case consists of two bundels:
series.json+ CMEC bundles) - tracked in git and part of the PR diff.Key decisions
test_case_version(int inmanifest.json) — an explicit bump is the re-baseline trigger; CI hard-fails a changed bundle with no bump.NativeStoreProtocol — public read / gated write; backend deferred (R2 to be used for a prototype).execution.jsonfrom #3 for baseline provenance once it lands.