UVH-IN4: a judged stage row says who judged it, and on what - #4493
UVH-IN4: a judged stage row says who judged it, and on what#4493chelojimenez wants to merge 8 commits into
Conversation
Two defects, both about a reader being unable to weigh an advisory verdict. EVERY JUDGED ROW SHIPPED WITH NO EVIDENCE. `judgeEvidenceFromVerdict` never populated `reasons`, so `boundedJudgeReasons` returned undefined on the judge path and `judgeObserved` / `judgePartial` / `judgeFailed` reached every surface as a bare verdict — a stage marked failed by a model, with nothing at all to say why. Its D7 sibling `metadataAttributionEvidenceFromVerdict` had populated reasons all along, so the two judges disagreed about how much they owed a reader. The row now carries, in order of directness: the judge's own rationale when one exists, and otherwise the numbers the verdict was decided from — "LLM judge scored 0.42 against a 0.7 threshold". The second is not a rationale and is not dressed up as one, but it tells a reader which side of the line the run fell and by how much, where before there was nothing. Rounded to two places because a model-supplied score arrives with float noise, which is displaying precision the verdict never had. A GAP NAMED RATHER THAN HIDDEN: the backend does not persist a scored case's rationale. `goalCompletion.ts` writes score, threshold, verdict and — in the `error` band only — the failure text, and drops the per-case `reason`. This reads `reason`/`reasons` anyway, so the day that changes the rationale appears with no second change here. Closing it properly is a backend write plus a deploy, recorded as follow-up rather than folded in. THE PROVENANCE IS NOW IN THE LABEL. These five reasons are the only ones in the vocabulary decided by a model rather than a deterministic rule, and "the judge scored inside the partial band" did not say so. They now read "the LLM judge …", and `judgePartial` spells out what the band means. The labels live in `decision-labels.ts`, which all four surfaces already read from, so one edit moves CLI, JSON, JUnit and HTML together — and the D9 cross-surface corpus proves they stayed in step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4jTtZJsDeaterEzKwpF2p
GitHub Actions
Inactive
dosubot
Bot
added
size:L
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_8cae7f5f-f27c-4049-a960-e8ed92e137ac) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a342ba467c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // (`0.42000000000000004`), and this line is read by a person. Rounding does | ||
| // not invent precision the verdict lacks — it stops displaying precision it | ||
| // never had. | ||
| const show = (n: number) => String(Number(n.toFixed(2))); |
There was a problem hiding this comment.
When the score and threshold differ by less than the two-decimal rounding precision, this formatter can contradict the verdict computed from the original values. For example, a partial score of 0.699 against a 0.7 threshold is displayed as “scored 0.7 against a 0.7 threshold,” while the accompanying label says the score was below the threshold. Preserve enough precision to distinguish the values, or format their relationship explicitly.
Useful? React with 👍 / 👎.
| threshold === undefined | ||
| ? `LLM judge scored ${show(score)}` | ||
| : `LLM judge scored ${show(score)} against a ${show( | ||
| threshold, | ||
| )} threshold`, |
There was a problem hiding this comment.
For partial and fail verdicts with a persisted partialFloor, the classification was decided using both the threshold and the partial floor, but this evidence reports only the threshold. For example, a score of 0.5 with threshold 0.7 and floor 0.6 is labeled as below the partial floor, yet the displayed numbers cannot explain why it is a failure rather than partial. Include the floor when it determines the band so the new provenance actually exposes the decision boundary.
Useful? React with 👍 / 👎.
Internal previewPreview URL: https://mcp-inspector-pr-4493.up.railway.app |
…s wording
The judge labels in this change gained "LLM" ("no judge verdict was ever owed"
became "no LLM judge verdict was ever owed"), and this test asserted the old
sentence verbatim, so it failed the moment the copy moved. CI here does not run
the client suite on pull requests, which is why it went unnoticed.
Fixed by asserting against STAGE_REASON_LABELS.judgeNotRequested rather than a
copy of its text. That is what the test's own name claims to check — the row
renders its reason's PUBLISHED label, which is what makes all four renderers
say the same thing — and it does not break again the next time the wording is
polished.
Still load-bearing: making the component render the raw wire enum instead of
the label fails exactly this test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4jTtZJsDeaterEzKwpF2p
GitHub Actions
Inactive
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_287471d0-9255-4e06-b2c8-c8591a6db18e) |
|
Pushed a fixup commit: this PR broke The test asserted the judge label's wording verbatim — The fix asserts against Full inspector suite on this branch after the fix: 10,969 passed, 29 skipped, 990 files. Generated by Claude Code |
…vc-mcp-eval-reporting-gyycwl-in4
GitHub Actions
Inactive
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_f0bdac33-d943-49e8-97df-514682373d6f) |
|
Correcting my comment above, and fixing the underlying cause. I said "CI here does not run the client suite on pull requests." That is wrong as stated — it runs a full matrix on PRs based on
So every PR in this stack above #4488 was green on previews and review bots alone. That is how the label break reached this PR unnoticed. The workflow already carries an allowlist pattern for an earlier stack, so there is a precedent rather than a new mechanism. I've added one for this stack at the root (#4488) and merged it forward through all seven branches — for Generated by Claude Code |
…vc-mcp-eval-reporting-gyycwl-in4
… onto one Two review findings, both about an evidence line failing to support the claim printed beside it — the one job evidence has. THE PARTIAL FLOOR WAS NEVER NAMED. A partial or fail band is settled by the threshold AND the partial floor together, but the line mentioned only the threshold: 0.5 against a 0.7 threshold with a 0.6 floor read as a plain miss and said nothing about why it failed rather than landing in the partial band. The floor is now named exactly when it is one of the numbers the decision turned on, and left out of a band it did not decide — a passing row should not carry a number that had no part in its outcome. (The band is `verdict.verdict`, not `verdict.status`; the first version of this fix read the wrong field and its test caught it.) ROUNDING COULD CONTRADICT THE BAND. At two decimals, 0.699 against a 0.7 threshold rendered as "scored 0.7 against a 0.7 threshold" beside a fail band: the numbers arguing with the label, with no way for a reader to tell which was wrong. Precision now grows only as far as it must — two decimals when the values are already distinguishable there, then the first deeper precision that keeps every DIFFERENT pair looking different, capped at six. Equal values still render equal, so a score exactly on its threshold reads that way, and 0.42000000000004 is still trimmed to 0.42, which is why the rounding existed at all. Mutation-checked: dropping the floor, naming it on every band, and reverting to fixed two-decimal rounding each fail exactly their intended tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4jTtZJsDeaterEzKwpF2p
GitHub Actions
Inactive
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_e430adda-fb22-4372-9d9a-b3c8c95aa55d) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 81a5b50a48
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const collides = bounds.some( | ||
| (bound) => bound !== score && at(bound, digits) === at(score, digits), | ||
| ); |
There was a problem hiding this comment.
When a failing verdict has distinct threshold and floor values that round alike while the score is farther away—for example, score 0.5, threshold 0.7001, and floor 0.6999—this loop stops at two decimals because it only compares each boundary with the score. The resulting evidence reports both a 0.7 threshold and a 0.7 partial floor, erasing the partial band's configured width. Fresh evidence beyond the prior score/threshold rounding comment is that the newly added second boundary is never compared with the first; choose precision using all pairwise-distinct displayed values.
Useful? React with 👍 / 👎.
|
Both Codex findings were right — each was an evidence line failing to support the claim printed beside it, which is the one job evidence has. Fixed in The partial floor was never named. A The floor is now named exactly when it is one of the numbers the decision turned on, and left out of a band it did not decide — a passing row should not carry a number that had no part in its outcome. (Worth noting: my first attempt read Rounding could contradict the band. Confirmed — at two decimals, 0.699 against a 0.7 threshold rendered as "scored 0.7 against a 0.7 threshold" beside a Precision now grows only as far as it must: two decimals when the values are already distinguishable there, then the first deeper precision that keeps every different pair looking different, capped at six — past that a judge score is float noise and an unreadable line explains nothing. Equal values still render equal, so a score exactly on its threshold reads that way, and Mutation-checked: dropping the floor, naming it on every band, and reverting to fixed two-decimal rounding each fail exactly their intended tests. Judge suite: 50 passed. Generated by Claude Code |
…vc-mcp-eval-reporting-gyycwl-in4
Follow-up to the precision fix, and a gap in it. It chose precision by comparing each boundary against the SCORE, never the boundaries against each other — so a fail verdict with a 0.7001 threshold and a 0.6999 partial floor rendered both as `0.7` while the score sat far from either, collapsing the partial band's configured width in the one line whose job is to show it. Precision is now chosen so every pairwise-distinct displayed value stays distinguishable. Equal values still render equal and the six-decimal cap still holds. Mutation-checked: reverting to the score-only comparison fails exactly the new test. Judge suite: 51 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4jTtZJsDeaterEzKwpF2p
GitHub Actions
Inactive
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_1d24a46a-3dcc-4461-9529-5e646ccd98db) |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Right again — a gap in my own precision fix. Fixed in It chose precision by comparing each boundary against the score, and never the boundaries against each other. So a Precision is now chosen so that every pairwise-distinct displayed value stays distinguishable. Equal values still render equal, and the six-decimal cap still holds. Mutation-checked: reverting to the score-only comparison fails exactly the new test. Judge suite: 51 passed. Generated by Claude Code |
…vc-mcp-eval-reporting-gyycwl-in4
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_eec792a7-9bb5-4dda-be10-be4d2c555f42) |
2 participants
UVH-IN4 — inspector lane, step 5. Presentation + evidence only; no analyzer bump. Stacked on #4492.
Two defects, both about a reader being unable to weigh an advisory verdict.
1. Every judged row shipped with no evidence
judgeEvidenceFromVerdictnever populatedreasons, soboundedJudgeReasonsreturnedundefinedon the judge path andjudgeObserved/judgePartial/judgeFailedreached every surface as a bare verdict — a stage marked failed by a model, with nothing at all to say why.Its D7 sibling
metadataAttributionEvidenceFromVerdicthad been populating reasons all along, so the two judges disagreed about how much they owed a reader.The row now carries, in order of directness:
LLM judge scored 0.42 against a 0.7 threshold.The second is not a rationale and is not dressed up as one, but it tells a reader which side of the line the run fell and by how much, where before there was nothing. Scores are rounded to two places: a model-supplied score arrives with float noise (
0.42000000000000004), and showing it is displaying precision the verdict never had.2. A gap named rather than hidden
The backend does not persist a scored case's rationale.
convex/goalCompletion.tswritesscore,threshold,verdictand — in theerrorband only — the failure text, and drops the per-casereasonthe judge produced. I verified this against the write itself rather than assuming.So path 1 above is dormant today. It reads
reason/reasonsanyway, so the day the backend persists one the rationale appears with no second change on this side, and the file says plainly that this is the situation. Closing it properly is a backend write plus a deploy — recorded as follow-up rather than folded into an inspector-only PR.3. The provenance is now in the label
These five reasons are the only ones in the vocabulary decided by a model rather than a deterministic rule, and "the judge scored inside the partial band" did not say so. A reader who cannot tell an assertion failure from an advisory verdict cannot weigh the row.
They now read "the LLM judge …", and
judgePartialspells out what the band actually means — above the floor, below the threshold.On centralization: the labels already live in
sdk/src/contract/decision-labels.ts, which the client presentation, CLI, JUnit and HTML renderers all read from. So this is one edit that moves four surfaces together, and the D9 cross-surface corpus proves they stayed in step — no new layer was needed, and adding one would have been the wrong move.Premise correction worth recording
The program plan described
judgePartialas "rendering as plain failure". It does not, and did not: all four renderers already print the distinct reason beside the state. Only the state badge readsfailed, which is correct — a partial is a failure of the threshold. So this PR is the realreasonsdefect plus copy that makes provenance explicit, and the PR body says that rather than claiming a rendering bug that was not there.Verification
errorband carries why the judge could not score (the one rationale the backend does persist); a scored row with no numbers stays bare rather than inventing one.sdk6891 passed; inspector server 704 passed; CLI 1182 passed.Generated by Claude Code
Note
Low Risk
Changes are eval presentation and copy on the judge second-pass evidence path, with broad test coverage and no analyzer or gating logic changes.
Overview
Judged user value stage rows no longer ship as bare verdicts:
judgeEvidenceFromVerdictnow fillsreasons—the judge’s own text when present (reason/reasons/error), otherwise a factual line likeLLM judge scored … against a … threshold, with the partial floor named only forpartial/failbands.Adaptive display precision (
showAgainst) picks 2–6 decimal places so scores never round onto a boundary they missed, pairwise-distinct threshold/floor/score values stay visible, float noise is trimmed, and ties (e.g. score exactly on threshold) still read equal.SDK
STAGE_REASON_LABELSfor the five model-decided judge reasons now say “LLM judge” and spell out what the partial band means; the session chain panel test assertsSTAGE_REASON_LABELSinstead of hardcoded copy. Presentation/evidence only on the second-pass path—scored-case rationale persistence remains a documented backend follow-up.Reviewed by Cursor Bugbot for commit 3e13d5f. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by cubic
Judge-scored stage rows now ship evidence instead of bare verdicts: the judge's own rationale when one exists, otherwise the numbers the verdict was decided from. The partial floor is named when it decided a
partial/failband and omitted when it didn't. Rendering precision grows only as deep as needed (up to six places) so no distinct value in the line collapses into another — a score never rounds onto a boundary it fell short of, and a narrow partial band's threshold and floor stay distinguishable — while float noise like0.42000000000000004still trims to0.42. Presentation only; no analyzer bump. Three patch changesets describe the evidence, precision, and provenance changes as released.The five model-decided reason labels now say "LLM judge", so readers can tell an advisory verdict from a deterministic rule failure. The reader accepts
reasonandreasons, though the backend doesn't persist a scored case's rationale yet — that write is tracked as follow-up.The chain panel test now asserts
STAGE_REASON_LABELSinstead of its wording, and new tests cover scored rows with numbers, the floor named in a band it decided and absent from one it didn't, boundary-neighbor precision, a score exactly on its threshold, an authored rationale winning, theerrorband, and a scored row with no numbers staying bare.Written for commit 3e13d5f. Summary will update on new commits.