Fix ground_truth.json resolution in tasks 012, 013, 014 oracles - #7
Open
henrydaum wants to merge 1 commit into
Open
Fix ground_truth.json resolution in tasks 012, 013, 014 oracles#7henrydaum wants to merge 1 commit into
henrydaum wants to merge 1 commit into
Conversation
Tasks 012, 013 and 014 resolve `ground_truth.json` from the *workspace*:
task_dir = w.parent.parent
Sandboxes are built at `<work_root>/<model_id>/<api_slug>/oc-bench-v2-.../workspace`,
so `w.parent.parent` is the api-slug directory in every configuration. The file
is never found, `gt_path.exists()` is False, and each oracle silently proceeds
with `gt = {}`.
`run_oracle` loads the oracle in place from `task.task_dir`, so
`Path(__file__).resolve().parent` is the task directory where the file actually
lives. 020-archive-checksum already resolves it this way.
Impact differs per task:
* **012-doc-synthesis — scores are stuck.** With `gt = {}` the three branches
disagree about what an empty expectation means: `trust_assessment` divides by
`len(expected)` and falls through to `else 0.0`, while `contradiction_detection`
and `report_quality` divide by empty lists and default to `1.0`. Every run
scores `0*0.25 + 1*0.35 + 1*0.40 = 0.75` regardless of what the agent wrote.
In the published results, **40 of 49 runs on this task are exactly 0.75**
(the rest are 0.0/0.35/0.4, where output files were missing). Replayed against
a real workspace, the score moves 0.7500 -> 0.4917 — and note the direction:
the broken oracle awards 0.0 on `trust_assessment`, the dimension that agent
actually did best at (0.86), and full credit on the two it did worst at.
* **014-task-decomposition — a check is silently skipped.** `expected_subtask_topics`
defaults to `[]`, so `if expected_topics:` is never true and topic coverage is
never scored, though `ground_truth.json` defines five topics. Published check
details record `{"topics_covered": 0, "total_topics": 0}` for every run.
* **013-image-edit — latent only.** Its `gt.get(...)` defaults happen to duplicate
`ground_truth.json` exactly, so no score changes today. Fixed for consistency,
and so the two copies cannot drift apart.
Verified by replaying all three oracles against real task workspaces: 012 changes
as above, 013 and 014 are unchanged on those particular runs.
Possible follow-up, not included here to keep this minimal: a missing
`ground_truth.json` currently re-weights a task instead of failing it. Raising
rather than defaulting to `{}` would turn a silent mis-scoring into a loud error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three task oracles resolve
ground_truth.jsonfrom the workspace instead of the task directory:Sandboxes are built at
<work_root>/<model_id>/<api_slug>/oc-bench-v2-.../workspace, sow.parent.parentlands on the api-slug directory in every configuration. The file is never found,exists()isFalse, and each oracle silently continues withgt = {}.run_oracleloads the oracle in place fromtask.task_dir, soPath(__file__).resolve().parentis the directory whereground_truth.jsonactually lives.020-archive-checksumalready resolves it this way — this PR just makes the other three consistent.Impact
012-doc-synthesis — scores are stuck regardless of agent output
With
gt = {}the three branches disagree about what an empty expectation means.trust_assessmentdivides bylen(expected)and falls through toelse 0.0;contradiction_detectionandreport_qualitydivide by empty lists and default to1.0. Every run lands on:This is visible in the published results — in
assets/leaderboard_scores.json, 40 of the 49 runs on this task score exactly 0.75. The remaining 9 are 0.0/0.35/0.4, cases where output files were missing entirely.Replaying the oracle against a real task workspace:
trust_assessmentcontradiction_detectionreport_qualityNote the direction: the broken path awards
0.0on the dimension this agent did best at, and full credit on the two it did worst at. The task currently cannot distinguish a good submission from a bad one.014-task-decomposition — a check is silently skipped
expected_subtask_topicsdefaults to[], soif expected_topics:is never true and topic coverage is never scored — althoughground_truth.jsondefines five topics. Recorded check details show{"topics_covered": 0, "total_topics": 0}on every run.013-image-edit — latent only
Its
gt.get(...)defaults happen to duplicateground_truth.jsonexactly, so no score changes today. Included so the two copies cannot drift apart, and so the file is actually read.Verification
All three oracles replayed against real task workspaces before and after. 012 changes as shown above; 013 and 014 are unchanged on those particular runs (014's bug is real but did not alter the score for an agent that happened to cover every topic).
Possible follow-up (not included here)
A missing
ground_truth.jsoncurrently re-weights a task rather than failing it, which is what let this stay invisible. Raising instead of defaulting to{}would turn a silent mis-scoring into a loud error. Happy to add that in a separate PR if you'd like it.