Run pre-edit verification, inject evidence packet, enforce no‑prose completion, and improve telemetry discovery#312
Open
mmprotest wants to merge 7 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Description
command,exit_code,timed_out,stdout_excerpt,stderr_excerpt, and parsed failure hints likefirst_failing_testanderror_summary(villani_code/state_runtime.py).build_task_evidence_packethelper that creates a compactTask evidence:packet with verification, failure excerpt, allowed edit paths, candidate files, and short guard instructions, and inject it into the first model turn when benchmark/verification/edit-scope metadata is present (villani_code/state_runtime.py,villani_code/state.py)._task_requires_patch()and_has_in_scope_patch()checks and block final completion with an actionable corrective message when no in-scope file changes occurred (bounded retries, returnsincomplete_no_patchon repeated failures) (villani_code/state.py)..villani_code/missions/*/runtime_events.jsonl,.villani_code/missions/*/tool_calls.jsonl,villani_debug/*/events.jsonl,villani_debug/*/tool_calls.jsonl) prefer recent artifacts by mtime windowing, parse JSONL robustly and skip bad lines, and include legacy fallback (villani_code/benchmark/agents/villani.py).tests/test_state_runtime.py(pre-edit verification runs even when expected files known, evidence packet contents) andtests/test_benchmark_agents.py(nested telemetry discovery). Files changed:villani_code/state_runtime.py,villani_code/state.py,villani_code/benchmark/agents/villani.py,tests/test_state_runtime.py,tests/test_benchmark_agents.py.Testing
pytest -q tests/test_state_runtime.py::test_fail_first_localization_runs_with_clear_expected_file tests/test_state_runtime.py::test_task_evidence_packet_contains_excerpt_and_scope tests/test_benchmark_agents.py::test_villani_run_agent_discovers_nested_telemetry_files; outcome: 3 passed.Codex Task