Skip to content

Match prompt-too-long errors by prefix so the no-diff retry can fire - #133

Open
blkdooGit wants to merge 1 commit into
anthropics:mainfrom
blkdooGit:fix/prompt-too-long-prefix-match
Open

blkdooGit wants to merge 1 commit into
anthropics:mainfrom
blkdooGit:fix/prompt-too-long-prefix-match

Conversation

@blkdooGit

Copy link
Copy Markdown

Summary

SimpleClaudeRunner.run_security_audit detects the prompt-too-long condition by comparing the result field against the exact string Prompt is too long. The API returns that marker with its token accounting appended, so the comparison never holds, PROMPT_TOO_LONG is never returned, and the retry-without-diff branch is unreachable.

Symptom

Verbatim from the result field of a production GitHub Actions run:

Prompt is too long · the request is ~1139713 tokens (limit 1000000) but the maximum allowed is 1000000

The scan reports success with zero findings. Nothing was analyzed.

Root cause

claudecode/github_action_audit.py:263

parsed_result.get('result') == 'Prompt is too long'

The marker is a prefix of the message, not the whole of it, so the equality is never true. PROMPT_TOO_LONG (l.264) is never returned, which makes the retry block at l.590-595 dead code.

What happens instead, in order:

  1. The condition falls through.
  2. Control reaches _extract_security_findings (l.274).
  3. 'result' in claude_output is true, so the error text is handed to the JSON parser.
  4. Parsing the error text as JSON fails.
  5. The empty structure is returned: findings: [], review_completed: False.
  6. run_security_audit returns success.

Fix

str(parsed_result.get('result', '')).startswith('Prompt is too long')

One line. The str(..., '') keeps the guard total for malformed payloads — the previous equality could never raise, and startswith on a non-string would.

Prefix rather than substring is deliberate: The output prompt is too long to display contains the phrase but is a different condition and must not trigger the retry.

Testing

Baseline before the change: 173 passed. After: 176 passed. Run as CI does, with pytest claudecode.

Three tests added to claudecode/test_claude_runner.py — this path had no coverage at all before:

  • the real message with token counts, verbatim
  • the bare Prompt is too long message, as a regression guard
  • four unrelated errors that must not trigger the retry: Rate limit exceeded, Invalid API key, overloaded_error, and The output prompt is too long to display

Demonstrated in red. Reverting the one-line fix while keeping the tests fails the first one, and the failure output shows the whole chain:

claudecode/test_claude_runner.py:283: in test_run_security_audit_prompt_too_long_with_token_counts
    assert success is False
E   assert True is False
------------------------------ Captured log call -------------------------------
ERROR    claudecode.json_parser:json_parser.py:88 Claude result text: Failed to parse JSON. Raw output: 'Prompt is too long · the request is ~1139713 tokens (limit 1000000) but the maximum allowed is 1000000'
=========================== short test summary ============================
FAILED claudecode/test_claude_runner.py::TestSimpleClaudeRunner::test_run_security_audit_prompt_too_long_with_token_counts
================== 1 failed, 2 passed, 22 deselected in 1.46s ==================

assert True is False is step 6; the log line above it is steps 3 and 4. The other two tests pass with and without the fix — they are regression guards, not the discriminator.

Relationship to #82 and #80

#82 takes a more complete approach to the 406 than anything here, and this PR does not touch that path — no overlap in the diffs. The two are complementary, and I think #82 needs this line to fully close #80.

With #82 merged and this line unchanged, the 406 stops killing the process but returns as the same silent zero through a different door:

  1. The fallback in Fix silent failure on large PRs (406 diff too large) #82 obtains the same large diff — it avoids the GitHub API limit, it does not reduce the diff's size.
  2. The Claude API responds with the prompt-too-long message shown above.
  3. The equality at l.263 does not match, so there is no retry without the diff.
  4. Control falls to _extract_security_findings: 'result' in claude_output is true, the JSON parse of the error message fails, and the empty structure is returned with findings: [] and review_completed: False.
  5. run_security_audit returns success.
  6. action.yml decides on findings_count and never reads review_completed.

The result is a green check with zero findings and no analysis performed — the symptom #80 reports.

Marked Refs #80 rather than Fixes #80, since closing that issue takes #82 as well.

Separate observation, not part of this PR

action.yml gates on findings_count and contains no reference to review_completed (zero matches in the file). Since _extract_security_findings sets review_completed: False on every failure path while still returning a zero count, any failure that yields zero findings is indistinguishable from a clean scan at the action level.

This PR fixes one route into that state. The general property — that the signal for "nothing was reviewed" exists in the payload but is never consumed by the action — seemed worth flagging separately for maintainers to weigh.

https://claude.ai/code/session_01VQyxybenLFB1GryQQ7XSmw

The check at github_action_audit.py:263 compared the result field against
the exact string 'Prompt is too long'. The API returns that marker with its
token accounting appended, for example:

    Prompt is too long · the request is ~1139713 tokens (limit 1000000) but
    the maximum allowed is 1000000

Because the equality never held, PROMPT_TOO_LONG was never returned and the
retry-without-diff branch was unreachable. The run instead fell through to
_extract_security_findings, which found a 'result' key, failed to parse the
error text as JSON, and returned the empty structure with review_completed
set to False while run_security_audit reported success.

Match on the prefix instead. The str() keeps the guard total for malformed
payloads: the previous equality could never raise, and startswith on a
non-string would.

Adds three tests: the real message with token counts, the bare message as a
regression guard, and four unrelated errors that must not trigger the retry.
One of them contains the phrase without beginning with it, which is why the
check matches a prefix rather than a substring.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Silent failure on large PRs: 406 diff too large swallowed as 0 findings

1 participant