[lanyue-llk]

Summary

Preserve SERP benchmark inputs and scoring after the Web Search package refactor. Compose the 149-scenario and 831-factual datasets through hash-bound references, preserving their order without copying the datasets.

Retain the frozen SERP candidate-packet format, explicit source eligibility rules and reference-answer citation cleanup, with paired scoring controls. Report instances, provider adapters, native response corrections and cumulative reference quarantine workflows belong to Browser Eval; this package supplies definitions and provider-independent scoring.

Validation

• make check: 60 tests passed, including accepted/rejected dataset compositions, source eligibility, citation handling and task acceptance.
• Python 3.11–3.14 CI passed.
• Browser Eval integration pins this commit and its dataset registry. Saved-response correction and sequential-quarantine scenarios run under feat(search): add reviewed SERP retrieval comparisons browser-eval#63.
• No provider or model calls.

Depends on #1. Browser Eval integration: lexmount/browser-eval#63.

[lanyue-llk]

Review conclusion

Fail — a new reference quarantine can silently remove an existing quarantine.

Purpose: preserve the extracted benchmark's SERP dataset composition, candidate packets and zero-call scoring corrections. Passing the native normalizer from the execution owner is a sound boundary; the package does not need provider execution code.

Findings and recommendations

One P1 defect is attached inline. Preserve prior exclusion decisions when extending the profile.

Validation: make check passed all 60 tests and lint/format checks. Built the wheel and loaded all 980 cases plus native-normalization regression definitions from an isolated installation outside the checkout. An additional sequential-quarantine scenario failed: excluding a, then excluding b, leaves only b excluded. No provider/model calls were made.

Companion review: lexmount/browser-eval#63.

[lanyue-llk]

[P1] Preserve existing quarantines when adding an adjudication

revised starts as a copy of the supplied profile, but these assignments replace its entire factual_quality_exclusions and reference_adjudications inventories with only this revision's entries. I reproduced two valid zero-call revisions: first quarantine factual case a, then quarantine b. The returned profile contains only ['b'], and a's supporting decision is gone. A report using that profile can put the known-defective reference a back into every method's quality denominator without any explicit readmission decision.

Merge new exclusions with existing ones and preserve the corresponding prior adjudications, validating duplicate/conflicting decisions. Test a profile that already has quarantined cases and a later correction adding a different case, including the resulting shared quality population.