Skip to content

Fix ground_truth.json resolution in tasks 012, 013, 014 oracles - #7

Open
henrydaum wants to merge 1 commit into
Qihoo360:mainfrom
henrydaum:fix/oracle-ground-truth-path
Open

Fix ground_truth.json resolution in tasks 012, 013, 014 oracles#7
henrydaum wants to merge 1 commit into
Qihoo360:mainfrom
henrydaum:fix/oracle-ground-truth-path

Conversation

@henrydaum

Copy link
Copy Markdown

Three task oracles resolve ground_truth.json from the workspace instead of the task directory:

task_dir = w.parent.parent          # tasks/012, 013, 014
gt_path  = task_dir / "ground_truth.json"
gt = json.loads(gt_path.read_text(encoding="utf-8")) if gt_path.exists() else {}

Sandboxes are built at <work_root>/<model_id>/<api_slug>/oc-bench-v2-.../workspace, so w.parent.parent lands on the api-slug directory in every configuration. The file is never found, exists() is False, and each oracle silently continues with gt = {}.

run_oracle loads the oracle in place from task.task_dir, so Path(__file__).resolve().parent is the directory where ground_truth.json actually lives. 020-archive-checksum already resolves it this way — this PR just makes the other three consistent.

Impact

012-doc-synthesis — scores are stuck regardless of agent output

With gt = {} the three branches disagree about what an empty expectation means. trust_assessment divides by len(expected) and falls through to else 0.0; contradiction_detection and report_quality divide by empty lists and default to 1.0. Every run lands on:

0.0*0.25 + 1.0*0.35 + 1.0*0.40 = 0.75

This is visible in the published results — in assets/leaderboard_scores.json, 40 of the 49 runs on this task score exactly 0.75. The remaining 9 are 0.0/0.35/0.4, cases where output files were missing entirely.

Replaying the oracle against a real task workspace:

upstream with this PR
trust_assessment 0.0 0.86
contradiction_detection 1.0 0.3333
report_quality 1.0 0.40
outcome_score 0.7500 0.4917

Note the direction: the broken path awards 0.0 on the dimension this agent did best at, and full credit on the two it did worst at. The task currently cannot distinguish a good submission from a bad one.

014-task-decomposition — a check is silently skipped

expected_subtask_topics defaults to [], so if expected_topics: is never true and topic coverage is never scored — although ground_truth.json defines five topics. Recorded check details show {"topics_covered": 0, "total_topics": 0} on every run.

013-image-edit — latent only

Its gt.get(...) defaults happen to duplicate ground_truth.json exactly, so no score changes today. Included so the two copies cannot drift apart, and so the file is actually read.

Verification

All three oracles replayed against real task workspaces before and after. 012 changes as shown above; 013 and 014 are unchanged on those particular runs (014's bug is real but did not alter the score for an agent that happened to cover every topic).

Possible follow-up (not included here)

A missing ground_truth.json currently re-weights a task rather than failing it, which is what let this stay invisible. Raising instead of defaulting to {} would turn a silent mis-scoring into a loud error. Happy to add that in a separate PR if you'd like it.

Tasks 012, 013 and 014 resolve `ground_truth.json` from the *workspace*:

    task_dir = w.parent.parent

Sandboxes are built at `<work_root>/<model_id>/<api_slug>/oc-bench-v2-.../workspace`,
so `w.parent.parent` is the api-slug directory in every configuration. The file
is never found, `gt_path.exists()` is False, and each oracle silently proceeds
with `gt = {}`.

`run_oracle` loads the oracle in place from `task.task_dir`, so
`Path(__file__).resolve().parent` is the task directory where the file actually
lives. 020-archive-checksum already resolves it this way.

Impact differs per task:

* **012-doc-synthesis — scores are stuck.** With `gt = {}` the three branches
  disagree about what an empty expectation means: `trust_assessment` divides by
  `len(expected)` and falls through to `else 0.0`, while `contradiction_detection`
  and `report_quality` divide by empty lists and default to `1.0`. Every run
  scores `0*0.25 + 1*0.35 + 1*0.40 = 0.75` regardless of what the agent wrote.
  In the published results, **40 of 49 runs on this task are exactly 0.75**
  (the rest are 0.0/0.35/0.4, where output files were missing). Replayed against
  a real workspace, the score moves 0.7500 -> 0.4917 — and note the direction:
  the broken oracle awards 0.0 on `trust_assessment`, the dimension that agent
  actually did best at (0.86), and full credit on the two it did worst at.

* **014-task-decomposition — a check is silently skipped.** `expected_subtask_topics`
  defaults to `[]`, so `if expected_topics:` is never true and topic coverage is
  never scored, though `ground_truth.json` defines five topics. Published check
  details record `{"topics_covered": 0, "total_topics": 0}` for every run.

* **013-image-edit — latent only.** Its `gt.get(...)` defaults happen to duplicate
  `ground_truth.json` exactly, so no score changes today. Fixed for consistency,
  and so the two copies cannot drift apart.

Verified by replaying all three oracles against real task workspaces: 012 changes
as above, 013 and 014 are unchanged on those particular runs.

Possible follow-up, not included here to keep this minimal: a missing
`ground_truth.json` currently re-weights a task instead of failing it. Raising
rather than defaulting to `{}` would turn a silent mis-scoring into a loud error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant