Repository navigation
fix(scoring): verify saved responses and revision evidence - #7
Open
lanyue-llk wants to merge 21 commits into
Open
lanyue-llk wants to merge 21 commits into
lanyue-llk wants to merge 21 commits into
Conversation
This was referenced Oct 20, 2026
lanyue-llk
commented
Oct 22, 2026
lanyue-llk
commented
Oct 22, 2026
lanyue-llk
commented
Oct 22, 2026
lanyue-llk
commented
Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and result
Live and saved responses could receive different verdicts, and a task revision could name regression controls without executing them. This PR gives Browser Eval and other callers the same deterministic scoring and revision-export contract.
Changes
inspect-revisionexposes those checks.The branch includes current main's article acceptance contracts. Expanded main-content references and unified mixed requirements remain in draft #8. Execution, retained evidence and all three report pipelines are handled by Browser Eval #56.
Validation
make check: 2,541 tests and 321 subtests passed after integrating current main.6bd263ed; the PR has no merge conflict.Historical report scope
The published 430-task report retains its frozen snapshot and scorer binding. Its complete comparison against the audited
e1b6efc1definitions contains 286 matching contracts, 91 retrieval-hint-only differences and 53 other differences. Original/revised-definition pass counts are 378/360/229/223/217/190 and 385/367/233/227/222/194 respectively. LZNOIP's eight canonical English navigation aliases are preserved by the report's bound revision. These saved results are not recomputed when this package's source identity changes.The full scope and source-copy audit are attached to the Notion review. The existing report keeps its normal version presentation and permissions.
Scoring and partial reruns