Skip to content

Phase 1 runner hardening: completion audit, schema-recovery, and shell guardrails#276

Open
mmprotest wants to merge 3 commits into
mainfrom
mmprotest/implement-phase-1-runner-hardening
Open

Phase 1 runner hardening: completion audit, schema-recovery, and shell guardrails#276
mmprotest wants to merge 3 commits into
mainfrom
mmprotest/implement-phase-1-runner-hardening

Conversation

@mmprotest

Copy link
Copy Markdown
Owner

Motivation

  • Improve runner execution discipline at the truthfulness boundaries without making the runner deterministic or task-specific.
  • Block obvious fake/completed claims, recover from repeated malformed structured tool calls, and reject shell forms that are clearly incompatible with the runtime environment.
  • Keep checks shallow, structural, low-assumption, and observable in runner traces.

Description

  • Add a structural completion audit module at villani_code/completion_audit.py that returns passed/failed, a concrete issues list, and a compact repair_brief built from observed workspace and transcript state.
  • Gate final completion in Runner.run() (both completion exit paths) by calling run_completion_audit(...); when the audit fails the run is not marked completed, a completion_audit_blocked event is emitted, and a concise factual repair brief is injected back to the agent for remediation (no fixed workflow enforced). (changes: villani_code/state.py, new villani_code/completion_audit.py).
  • Add runtime tool-call schema-collapse recovery in Runner: track recent malformed schema failures per tool, trigger a short schema-focused corrective injection after repeated invalid calls, emit visible events (tool_schema_recovery_triggered, tool_schema_degradation_escalated) and escalate if failures persist, and clear degraded state after a successful call (changes: villani_code/state.py).
  • Add environment-aware shell-form guardrails: validate_shell_form() in villani_code/shells.py detects clearly incompatible patterns (Windows heredoc/redirection, POSIX $?, impractically large inline commands) and _run_bash() in villani_code/tools.py rejects such forms before execution while emitting a command_rejected debug callback; debug recording of command_rejected was added to Runner for traceability (changes: villani_code/shells.py, villani_code/tools.py, villani_code/state.py).
  • Added focused tests that exercise the new behaviors (new file tests/test_runner_phase1_hardening.py) and left existing tool/schema/tests passing.

Testing

  • Ran pytest -q tests/test_runner_phase1_hardening.py and observed 3 passed (completion audit blocking + repair, repeated malformed tool-call recovery message, Windows-incompatible shell rejection).
  • Ran pytest -q tests/test_tools_schema.py tests/test_tools_hardening.py tests/test_trace_runtime_path.py and observed 12 passed (no regressions in tool schema or trace/runtime path behavior).
  • All new tests and the exercised existing tests passed in this run, and debug/trace events for audit/recovery/guardrails are emitted for visibility.

Codex Task

…on-audit

Remove Phase-1 hard completion audit gate (preserve tool-call recovery & shell guardrails)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant