Skip to content

fix(ai-hook): say when an MCP tool call was not checked against policy - #1391

Open
amascia-gg wants to merge 3 commits into
mainfrom
amascia/mcp-policy-not-checked-warning
Open

fix(ai-hook): say when an MCP tool call was not checked against policy#1391
amascia-gg wants to merge 3 commits into
mainfrom
amascia/mcp-policy-not-checked-warning

Conversation

@amascia-gg

Copy link
Copy Markdown
Member

Context

send_mcp_activity() returned MCPActivityResponse(allowed=True, reason="") on every failure path — byte-identical to a genuine policy approval. A timeout, a 500, a malformed response, or a bug in parse_mcp_activity all permitted the MCP tool call, and the caller had no way to tell them from "the API said yes".

So an unreachable policy endpoint silently allows every MCP tool call. Same class of invisible fail-open as #1375 (heredoc scans abandoned) and #1378 (keychain write-probe).

What has been done

send_mcp_activity() now returns Optional[MCPActivityResponse], where None means "no answer obtained". The caller fails open with a warning, reusing what the secret scan already does when it cannot scan (HookResult.allow_with_warningsystemMessage).

Behaviour stays fail-open — a network blip must never block a tool call. The change is visibility, not denial.

Three related fixes on the way through:

  • The "not an MCP pre-tool-use" early return stays a genuine, silent allow. Warning there would fire on every Read, Bash and Edit call.
  • refresh_and_maybe_submit_discovery() moves inside the try block. It walks the filesystem and calls the API, so it fails for the same reasons the request does. Outside, its failures bypassed the MCP fail-open and surfaced as the generic "could not scan for secrets" warning — the wrong message, since the scan itself was fine.
  • The bare except Exception is split: an expected RequestException logs at debug, anything else at warning, so a bug is findable instead of looking like a network blip.

One bug found while wiring it up: _scan_payloads() only returns a result for a blocking payload and otherwise builds a fresh HookResult.allow(), which dropped the warning. It now collects non-blocking warnings and attaches them to the final allow.

Validation

  • pytest tests/unit2603 passed, 4 skipped
  • 7 new tests: API error / network error / unexpected error / discovery failure all yield no answer; a non-MCP payload stays a silent allow; a real allow warns about nothing; and the warning survives _scan_payloads
  • Revert check: with the source reverted and the tests kept, 4 of the new tests fail — so they catch the regression rather than restating the code
  • black / isort / flake8 clean

Not covered: the Rust ai-hook port (#1381) does not implement MCP policy at all, so this widens the Python/Rust behavioural gap. The equivalence harness has no MCP cases yet — tracked separately.

PR check list

  • Tests included
  • Changelog entry (changelog.d/)

send_mcp_activity() returned MCPActivityResponse(allowed=True) on every failure
path, which is byte-identical to a genuine policy approval. So a timeout, a 500,
a malformed response, or a bug in parse_mcp_activity all permitted the MCP tool
call with nothing to distinguish them from "the API said yes" -- an unreachable
policy endpoint silently allowed everything.

It now returns None for "no answer obtained", and the caller fails open *with a
warning*, reusing the machinery the secret scan already uses when it cannot scan
(HookResult.allow_with_warning -> systemMessage). Behaviour stays fail-open: a
network blip must never block a tool call.

Three related fixes on the way:

- The "not an MCP pre-tool-use" early return stays a genuine, silent allow.
  Warning there would fire on every Read, Bash and Edit call.
- refresh_and_maybe_submit_discovery() moves inside the try block. It walks the
  filesystem and calls the API, so it fails for the same reasons the request
  does; outside, its failures bypassed the MCP fail-open and surfaced as the
  generic "could not scan for secrets" warning, which was the wrong message --
  the secret scan itself was fine.
- The bare `except Exception` is split: expected RequestException logs at debug,
  anything else logs at warning, so a bug is findable instead of looking like a
  network blip.

_scan_payloads() only returned a result for a *blocking* payload and otherwise
built a fresh HookResult.allow(), which dropped the warning. It now collects
non-blocking warnings and attaches them to the final allow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@amascia-gg
amascia-gg requested review from a team as code owners August 5, 2026 13:37
@codecov

codecov Bot commented Aug 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.03%. Comparing base (8d7e53d) to head (0663812).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1391   +/-   ##
=======================================
  Coverage   94.02%   94.03%           
=======================================
  Files         199      199           
  Lines       12286    12304   +18     
=======================================
+ Hits        11552    11570   +18     
  Misses        734      734           
Flag Coverage Δ
unittests 94.03% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@amascia-gg amascia-gg self-assigned this Aug 6, 2026
amascia-gg and others added 2 commits August 11, 2026 17:46
Cut the multi-paragraph narration around send_mcp_activity's fail-open
down to the why, and shorten the user-facing warning to two sentences.
The "(it was still scanned for secrets)" claim is dropped: the scan is
skipped for an MCP call with no arguments, and yields no result when the
API answers the scan with an error, so the claim was not always true.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The warning fired on any non-2xx, including the 404 an older self-hosted
instance returns for the new /v1/agent-activity/mcp-activity route and the
4xx a workspace without the feature gets. Those users would see it on every
MCP tool call, forever, so they would stop reading it.

A 4xx is now a silent allow: the instance answered, it just has no policy to
offer. Only an unreachable server or a 5xx warns. There is no capability flag
in the cached instance metadata to gate on, hence the status split.

The warning also survives a blocking secret scan, and no longer names a
command to run: it reaches the model too (Cursor's agent_message), which
would run it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant