Fail loud when false-positive filtering is unavailable (remove the fixed-model probe)

[simonstudios]

The defect

FindingsFilter.__init__ opens with ClaudeAPIClient.validate_api_access(), whose probe call hard-codes model="claude-3-5-haiku-20241022" (claudecode/claude_api_client.py:62 on main). That id is not settable by any action input — claude-model sets the scan model only.

claude-3-5-haiku-20241022 retired on 2026-02-19. Since then, on every run of this action:

1. the probe raises not_found_error;
2. findings_filter.py:187-192 catches it, logs a warning, sets use_claude_filtering = False and drops the client;
3. filter_findings takes the else branch and stamps every finding confidence_score: 10.0, justification: 'Claude filtering disabled';
4. github_action_audit.py emits that through the normal success path.

The scan still runs. Only the false-positive filter stops — and the output is indistinguishable in shape from output the filter actually passed. A fleet running this action logged 36 failed probe requests in a single day, and found out only because Anthropic emailed about the failed calls.

constants.py:8's DEFAULT_CLAUDE_MODEL fallback has the same problem: claude-opus-4-1-20250805 reached its published retirement date on 2026-08-05.

Why not just bump the probe's model id

That moves the identical trap to the next retirement. The probe's only job is to find out whether the API works — which the filter's own first call already does, against the model actually configured.

The change

• Delete validate_api_access() and the startup probe. The filter's first real call is the validation, and it uses the configured model.
• Add ClaudeFilteringUnavailableError, raised on a configuration-class API error — unknown/retired model id, bad or revoked key, no access to the model. Raised on the first failure with no retries: retrying an unknown model id only multiplies failed requests.
• github_action_audit.py exits EXIT_CONFIGURATION_ERROR with {"error": "FALSE_POSITIVE_FILTER_UNAVAILABLE: ...", "filter_unavailable": true}, so a caller can distinguish "the filter could not run" from a normal result.
• Refresh ids at their retirement date: DEFAULT_CLAUDE_MODEL → claude-opus-5, plus the example in action.yml and the documented default in README.md.

The classifier is deliberately narrow — not_found_error, authentication_error, permission_error, invalid-api-key, and Error code: 401/403/404. Rate limits, overload, timeouts and spend/usage-cap 400s are not configuration errors and still degrade gracefully through the existing retry and per-finding fallback paths. That matters: a workspace hitting its usage cap should not start failing security jobs.

Tests

claudecode/test_filter_fail_loud.py (16 cases) covers both directions — a not_found_error raises after exactly one call and names the configured model; rate-limit and spend-cap errors still retry and still keep findings; construction makes zero API calls; validate_api_access is gone.

pytest claudecode/ : 173 passed. (test_eval_engine.py's 11 failures are pre-existing on unmodified main in this environment — it writes to a hard-coded ~/code path.)

Note for maintainers

We are running this as a fork pin at simon-studios/claude-code-security-review until this lands, and will drop the fork when it does.

[sylvesterkaczmarek]

After the retry budget is exhausted, rate-limit/overload/spend-cap/connection failures still return success=False, and filter_findings() keeps the finding with confidence 10.0 and returns overall success. That means a run where Claude filtering never succeeded can still emit unfiltered results as normal filtered output. Please mark filtering incomplete/fail after retries are exhausted rather than silently promoting those findings.

[simonstudios]

Pushed two more commits after review, both closing gaps in the original change rather than adding scope. Updating the PR description would have buried them, so they are here.

1. action.yml swallowed the auditor's exit code, so the fix did not actually fail anything.

python -u claudecode/github_action_audit.py … || CLAUDECODE_EXIT_CODE=$? followed by only a ::warning:: makes sys.exit(EXIT_CONFIGURATION_ERROR) inert — the composite step exits 0 with findings_count=0 and the job stays green. So "fail loud when filtering is unavailable" held only for callers that parse the results JSON themselves; for everyone else the behaviour was unchanged from the bug.

Exit 2 now fails the step. Exit 1 stays swallowed — that is the auditor's "found a HIGH severity finding" signal, which is a result, not a failure, and callers gate on it themselves. The failure is raised after the outputs and workspace copies, so artifact upload and PR commenting still see everything.

2. A total transient filter failure was invisible.

On a persistent 429/529/timeout, every finding took the per-finding fallback, got justification: "Claude API failed: …", and filter_findings returned success. That output is indistinguishable from a filtered pass — which is exactly the state ClaudeFilteringUnavailableError was added to prevent, reproduced one branch over.

Now: claude_api_failures is counted and reported in filtering_summary.filter_analysis, so a partial failure is visible instead of assumed away; and when every finding failed — the filter produced no verdict at all — it raises.

The line is drawn at "nothing was filtered", not at "an error occurred", deliberately. A transient blip must not fail a whole security job; a filter that returned no verdicts did not run, whatever the reason. Transient errors still go down the retry path, and the configuration classifier stays as narrow as before — spend-cap 400s, 429s, 529s and timeouts are all still non-configuration.

One thing worth flagging for reviewers: the raise was first written one block too far down and landed inside the else: branch (filtering disabled), where the counter is always zero, so it never fired. The new tests caught it; a test that only asserted "no exception on the happy path" would not have.

pytest claudecode/ : 175 passed on this branch.