Skip to content

Run pre-edit verification, inject evidence packet, enforce no‑prose completion, and improve telemetry discovery#312

Open
mmprotest wants to merge 7 commits into
mainfrom
mmprotest/improve-runner-and-adapter-for-evidence-based-tasks
Open

Run pre-edit verification, inject evidence packet, enforce no‑prose completion, and improve telemetry discovery#312
mmprotest wants to merge 7 commits into
mainfrom
mmprotest/improve-runner-and-adapter-for-evidence-based-tasks

Conversation

@mmprotest

Copy link
Copy Markdown
Owner

Motivation

  • Make the runner more evidence-driven by always collecting pre-edit verification evidence when a visible verification command is present, so the model cannot infer fixes without checking the verification signal first.
  • Reduce premature prose-only completions for patch-required tasks by enforcing a lightweight completion guard that requires an in-scope repo change.
  • Improve telemetry ingestion for external adapters so nested debug/mission JSONL artifacts are discovered and malformed lines don't crash ingestion.

Description

  • Always run visible pre-edit verification once when a visible verification command exists and capture a bounded evidence record including command, exit_code, timed_out, stdout_excerpt, stderr_excerpt, and parsed failure hints like first_failing_test and error_summary (villani_code/state_runtime.py).
  • Add build_task_evidence_packet helper that creates a compact Task evidence: packet with verification, failure excerpt, allowed edit paths, candidate files, and short guard instructions, and inject it into the first model turn when benchmark/verification/edit-scope metadata is present (villani_code/state_runtime.py, villani_code/state.py).
  • Add a generic completion guard for patch-required tasks by adding _task_requires_patch() and _has_in_scope_patch() checks and block final completion with an actionable corrective message when no in-scope file changes occurred (bounded retries, returns incomplete_no_patch on repeated failures) (villani_code/state.py).
  • Improve adapter telemetry discovery in the Villani benchmark agent to scan nested locations (.villani_code/missions/*/runtime_events.jsonl, .villani_code/missions/*/tool_calls.jsonl, villani_debug/*/events.jsonl, villani_debug/*/tool_calls.jsonl) prefer recent artifacts by mtime windowing, parse JSONL robustly and skip bad lines, and include legacy fallback (villani_code/benchmark/agents/villani.py).
  • Add focused tests covering the new behaviors and helpers: tests/test_state_runtime.py (pre-edit verification runs even when expected files known, evidence packet contents) and tests/test_benchmark_agents.py (nested telemetry discovery). Files changed: villani_code/state_runtime.py, villani_code/state.py, villani_code/benchmark/agents/villani.py, tests/test_state_runtime.py, tests/test_benchmark_agents.py.

Testing

  • Ran targeted pytest selection: pytest -q tests/test_state_runtime.py::test_fail_first_localization_runs_with_clear_expected_file tests/test_state_runtime.py::test_task_evidence_packet_contains_excerpt_and_scope tests/test_benchmark_agents.py::test_villani_run_agent_discovers_nested_telemetry_files; outcome: 3 passed.
  • These tests validate (1) pre-edit verification is executed and captured even when expected files are present, (2) the evidence packet contains failure excerpt and allowed edit paths, and (3) adapter discovers nested telemetry JSONL files and surfaces their events.
  • Remaining limitations: the change is intentionally surgical and does not yet implement the full post-patch repair-loop gate or the broader generalized new-file rejection messaging for all cases; these follow-ups were left out to keep the change minimal and focused.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant