Skip to content

fix: use configured model in validate_api_access instead of retired claude-3-5-haiku - #128

Open
dcyfr wants to merge 1 commit into
anthropics:mainfrom
dcyfr:fix-validate-api-access-retired-model
Open

dcyfr wants to merge 1 commit into
anthropics:mainfrom
dcyfr:fix-validate-api-access-retired-model

Conversation

@dcyfr

@dcyfr dcyfr commented Aug 29, 2026

Copy link
Copy Markdown

Fixes #127.

validate_api_access() hardcoded claude-3-5-haiku-20241022 for its 10-token health check. That model was retired on 2026-02-19, so the check now fails unconditionally — the claude-model input never reaches it — and FindingsFilter responds by silently disabling Claude-powered false-positive filtering (findings_filter.py logs a warning and sets use_claude_filtering = False) while the run stays green.

This changes the health check to use self.model, the model the client was constructed with and will use for the actual filtering calls, so it validates the same access it is guarding.

Not included here (see #127): DEFAULT_CLAUDE_MODEL in constants.py falls back to claude-opus-4-1-20250805, which is also retired — happy to bump that in this PR too if you'd like.

🤖 Generated with Claude Code

…de-3-5-haiku

validate_api_access() hardcoded claude-3-5-haiku-20241022 for its 10-token
health check. That model was retired on 2026-02-19, so the check now fails
unconditionally regardless of the claude-model input, and FindingsFilter
responds by silently disabling Claude-powered false-positive filtering
(findings_filter.py logs a warning and sets use_claude_filtering=False)
while the action run stays green.

Use self.model — the model the client was constructed with and will use
for the actual filtering calls — so the health check validates the same
access it is guarding.
alexhansen-pointstire added a commit to PointS-Dev-Team/claude-code-security-review that referenced this pull request Sep 1, 2026
… 3.5

validate_api_access() was pinned to claude-3-5-haiku-20241022, retired on
2026-02-19. Every preflight 404'd with not_found_error, and findings_filter.py
treats a failed preflight as "Claude filtering unavailable" — it logs a warning
to stderr and sets use_claude_filtering = False. In a GitHub Action that stderr
goes to claudecode-error.log, which is never printed, so false-positive
filtering has been off since the retirement date with no visible signal.

Validating with self.model checks the model the client will actually call, so a
preflight failure now means something real rather than a stale constant.

Upstream: anthropics#127 (open, along with anthropics#69, anthropics#73,
anthropics#88, anthropics#90, anthropics#102, anthropics#103, anthropics#114, anthropics#123, anthropics#128 for the same defect).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alexhansen-pointstire

Copy link
Copy Markdown

Supporting this one — it matches what we independently diagnosed, and I want to add the impact detail plus a second defect that this PR alone doesn't cover.

The failure is swallowed, which is why it went unnoticed for six months

claude-3-5-haiku-20241022 was retired 2026-02-19. The preflight has 404'd on every run since. But findings_filter.py:188-192 treats a failed preflight as "Claude filtering unavailable" rather than an error:

valid, error = self.claude_client.validate_api_access()
if not valid:
    logger.warning(f"Claude API validation failed: {error}")
    self.claude_client = None
    self.use_claude_filtering = False

In the GitHub Action, that warning goes to claudecode-error.log, which action.yml prints only when the scan itself fails. A successful scan therefore reports ClaudeCode scan completed successfully and posts findings with false-positive filtering silently off — no error, no warning, nothing in the run log. That is the actual cost of this bug, and I don't think any of the ten open reports (#69, #73, #88, #90, #102, #103, #114, #123, #127, and this one) state it: they read as "a ping 404s," which sounds cosmetic.

We only found it because Anthropic's model-retirement notice reported 15 failed requests on 2026-08-26 from a key whose only user is CI, and none of our own code referenced the model. Every one of those 15 was a security review that produced findings and filtered none of them.

DEFAULT_CLAUDE_MODEL is also retired, so this fix alone isn't enough

constants.py:8:

DEFAULT_CLAUDE_MODEL = os.environ.get('CLAUDE_MODEL') or 'claude-opus-4-1-20250805'

claude-opus-4-1-20250805's retirement date (2026-08-05) has also passed. initialize_findings_filter constructs FindingsFilter without a model= argument, so any caller that doesn't set the claude-model input gets that default — and after this PR, validate_api_access would validate against it and 404 again. Same silent disable, different model ID. It also reaches the scan itself via --model DEFAULT_CLAUDE_MODEL (github_action_audit.py:227).

Suggest pairing this with a bump of that fallback to a current model, so an unset CLAUDE_MODEL degrades to "wrong tier" rather than "404".

Reproduction

Constructing FindingsFilter the way initialize_findings_filter does, with a stubbed client that 404s retired IDs the way the API does:

current main              preflight=claude-3-5-haiku-20241022  filtering_enabled=False
+ this PR's change        preflight=claude-opus-5              filtering_enabled=True

Holds with CLAUDE_MODEL set, empty, and unset. The main row is the control — it confirms the check detects the defect rather than merely passing.

We're carrying both changes on a fork for now. Happy to open a follow-up PR for the constants.py half if that's useful, or fold it in here — your call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

validate_api_access() hardcodes retired claude-3-5-haiku-20241022 — silently disables FP filtering since 2026-02-19

3 participants