Harden validation evidence handling#387
Open
mmprotest wants to merge 49 commits into
Open
Conversation
…tion-for-villani-code Add Pi bridge integration
…lures-in-pi-villani Fix Windows test failures by removing .cmd fixtures and adding test bridge injection
…proval-handling-and-ui-feedback Allow ExecutionPlan at policy layer and add visible Pi approval/run UI
…pproval-hang-issue
…execution-hang-issue Add Pi bridge approval diagnostics and confirm smoke test
Pi integration
Updated README to enhance clarity on Villani Code's capabilities and benchmark results.
…tch-to-version-1.7 Fix regular trace ATIF trajectory shape
…-for-terminal-bench-compatibility Make debug artifacts ATIF-compatible
…and-atif-v1.7-artifacts Add ATIF v1.7 and full transcript artifacts for regular TRACE debug runs
…er-execution-context-isolation Isolate task execution and retain cumulative validation evidence
…detection-in-runner Make no-progress detection failure-sensitive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Description
-o pipefail -cwhenbashis available, falling back to the existing shell behavior otherwise (TaskExecutionContext._run_process)._VALIDATION_INTENT_REand_WEAK_VALIDATION_RE, and exposeis_validation_likeandis_weakened_validation_commandhelpers.weakened_validation,changed_files_fingerprint,output_excerpt, andsteptoCommandRecord, add aweakenedflag toValidationEvidence, and downgrade/label weakened evidence in the model-visible observation.UnresolvedValidationFailureandSuccessfulValidationStatedataclasses and newAttemptStatefields to track unresolved validation failures and the last successful validation state, plustrack_validationandfinalization_warningslogic to record/clear failures and produce compact warnings.Runner._finish_bounded).Testing
tests/test_validation_evidence_hygiene.pyand ran it (all tests in that file passed).pytest -q tests/test_validation_evidence_hygiene.py tests/test_execution_context_isolation.py tests/test_evidence.py tests/test_loop.py tests/test_debug_mode_and_recorder.pywith success (76 passed).ruffandmypyon modified files which passed, and ran the full test suite wherepytest -qreturned639 passed, 11 skippedwith7unrelated pre-existing failures in planning/TUI/mission-state tests not caused by these changes.false | tailreturns nonzero whenbashis available, filtered commands are marked as weak, unresolved failed validation entries are recorded and cleared by later successful validation after relevant file changes, and a drift warning is emitted when files changed after the last successful validation.Codex Task