Skip to content

Harden validation evidence handling#387

Open
mmprotest wants to merge 49 commits into
mainfrom
mmprotest/implement-validation-evidence-handling-fixes
Open

Harden validation evidence handling#387
mmprotest wants to merge 49 commits into
mainfrom
mmprotest/implement-validation-evidence-handling-fixes

Conversation

@mmprotest

Copy link
Copy Markdown
Owner

Motivation

  • Prevent shell pipelines from masking failing commands and make validation evidence hygiene explicit to avoid weak/misleading success signals.
  • Provide minimal, generic tracking of validation-like commands so failed checks cannot silently disappear and file-drift after a last-successful validation is detectable.

Description

  • Run model-issued shell commands through Bash with -o pipefail -c when bash is available, falling back to the existing shell behavior otherwise (TaskExecutionContext._run_process).
  • Add detection for validation-intent and weak/masking patterns via _VALIDATION_INTENT_RE and _WEAK_VALIDATION_RE, and expose is_validation_like and is_weakened_validation_command helpers.
  • Extend telemetry and evidence structures by adding weakened_validation, changed_files_fingerprint, output_excerpt, and step to CommandRecord, add a weakened flag to ValidationEvidence, and downgrade/label weakened evidence in the model-visible observation.
  • Introduce UnresolvedValidationFailure and SuccessfulValidationState dataclasses and new AttemptState fields to track unresolved validation failures and the last successful validation state, plus track_validation and finalization_warnings logic to record/clear failures and produce compact warnings.
  • Wire compact finalization warnings into the existing finalization flow by appending the compact warning text to the final response without blocking finalization (Runner._finish_bounded).
  • Add focused tests and a changelog entry documenting the changes.

Testing

  • Added tests/test_validation_evidence_hygiene.py and ran it (all tests in that file passed).
  • Ran focused regression tests pytest -q tests/test_validation_evidence_hygiene.py tests/test_execution_context_isolation.py tests/test_evidence.py tests/test_loop.py tests/test_debug_mode_and_recorder.py with success (76 passed).
  • Ran lint/type checks with ruff and mypy on modified files which passed, and ran the full test suite where pytest -q returned 639 passed, 11 skipped with 7 unrelated pre-existing failures in planning/TUI/mission-state tests not caused by these changes.
  • Tests confirm false | tail returns nonzero when bash is available, filtered commands are marked as weak, unresolved failed validation entries are recorded and cleared by later successful validation after relevant file changes, and a drift warning is emitted when files changed after the last successful validation.

Codex Task

mmprotest and others added 30 commits May 18, 2026 16:29
…tion-for-villani-code

Add Pi bridge integration
…lures-in-pi-villani

Fix Windows test failures by removing .cmd fixtures and adding test bridge injection
…proval-handling-and-ui-feedback

Allow ExecutionPlan at policy layer and add visible Pi approval/run UI
…execution-hang-issue

Add Pi bridge approval diagnostics and confirm smoke test
Benchmark Bot and others added 19 commits May 31, 2026 09:58
Updated README to enhance clarity on Villani Code's capabilities and benchmark results.
…tch-to-version-1.7

Fix regular trace ATIF trajectory shape
…-for-terminal-bench-compatibility

Make debug artifacts ATIF-compatible
…and-atif-v1.7-artifacts

Add ATIF v1.7 and full transcript artifacts for regular TRACE debug runs
…er-execution-context-isolation

Isolate task execution and retain cumulative validation evidence
…detection-in-runner

Make no-progress detection failure-sensitive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant