Skip to content

fix: retain benchmark outcomes with measurement warnings - #3683

Merged
janicduplessis merged 2 commits into
mainfrom
fix/3679-benchmark-measurement-warnings
Oct 13, 2026
Merged

janicduplessis merged 2 commits into
mainfrom
fix/3679-benchmark-measurement-warnings

Conversation

@janicduplessis

@janicduplessis janicduplessis commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Description

A successful benchmark can be discarded because its build is slow, cache counters are missing, or diagnosis token usage is unavailable. Filtering those runs hides regressions and confuses missing telemetry with a failed task. Fixes #3679.

Solution

Keep proven outcomes valid and expose measurement warnings in run records, reports, and the viewer. Unknown timings and diagnosis costs remain unknown. Timeouts, device identity, evidence hashes, visual proof, source repair, and capture-before-source checks remain validity requirements.

Benign setup warnings now publish after evidence checks pass; source/evidence corrections and auxiliary sessions still require review. This deliberately changes the abnormal-build timing gate from #435 to retain regressions, and narrows #513's setup-review requirement. Historical rejected records are not silently rewritten.

Test plan

  • Focused guard, exporter, and viewer tests cover slow valid runs, missing counters, null diagnosis usage/cost, warning display, and benign setup publication. Existing tampered-evidence, incorrect-metric, device, and source-order refusals remain covered.
  • Replayed the accepted Android crash record: 19.169 s platform time, artifact hit, and three visible setup warnings; original record unchanged.
  • All repository checks pass (8,905 unit tests, published runtime, 115 e2e); final focused tests pass (187). Website production build passes. Website typecheck has the same 31 core-file errors on the unchanged base.
  • Visual check below uses synthetic warning data in the disposable built viewer; published benchmark records are unchanged.

Synthetic warning display

@janicduplessis
janicduplessis marked this pull request as ready for review October 13, 2026 12:13
@janicduplessis
janicduplessis added this pull request to the merge queue Oct 13, 2026
Merged via the queue into main with commit 9b34ea5 Oct 13, 2026
18 of 19 checks passed
@janicduplessis
janicduplessis deleted the fix/3679-benchmark-measurement-warnings branch October 13, 2026 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Separate benchmark validity from performance and measurement warnings

1 participant