[BUG] The review check has passed without reviewing since 2026-08-03 — workflow validation skips but exits success

[erikdarlingdata]

Component

CI

Describe the Bug

The review check has been passing without reviewing anything since 2026-08-03. It reports conclusion=success while the action skips, so every PR merged since then looked reviewed and wasn't.

This is self-inflicted and I caused it. Sequence:

1. claude-code-action validates that the workflow file it's running from is identical to the copy on the repository's default branch (main here). If it isn't, it skips — and exits success, not failure:

   ##[warning]Skipping action due to workflow validation: Workflow validation failed.
   The workflow file must exist and have identical content to the version on the
   repository's default branch.
   ...
   Exiting due to workflow validation skip
   ##[end-action id=...run;outcome=success;conclusion=success;duration_ms=1724]
   

2. On 2026-08-03 Skip the AI review job on Dependabot PRs #423 merged the Dependabot-skip guard into .github/workflows/claude-code-review.yml on dev. main still has the 2026-07-26 version ("Skip review on release PRs; widen the reviewer tool allowlist"). The only difference is the 8-line dependabot comment plus github.actor != 'dependabot[bot]' &&.

3. Every PR targets dev. So the running workflow never matches main, validation fails, and the action skips — on every PR, silently, green.

Confirmed on two PRs opened today (#426 and #427): both review jobs completed "successfully" in 12-13 seconds, versus the ~4-5 minutes a real review of the same size takes in the sibling repo, and both logs contain the validation-failure warning. Neither produced a review.

The tell is the duration. A green review check that finishes in ~12s has not read anything.

Expected Behavior

Either the review actually runs on PRs to dev, or the check fails loudly when it can't. A skip that reports success is worse than no check, because it manufactures confidence.

Suggested Fix

Sync main's copy of claude-code-review.yml with dev's so validation passes. Two ways:

1. Cherry-pick the workflow file onto main directly — fixes it immediately, no release needed. My recommendation, since the change is a comment plus one if: condition with no runtime effect on main.
2. Wait for the next dev → main release, which syncs it as a side effect. Correct but leaves the gate dead until then.

Worth knowing either way: PRs that themselves modify workflow files will always skip review under this validation model, which is by design and called out in the action's own message. #427 touches three workflow files, so its review would skip even with main synced. That's acceptable as long as it's understood rather than mistaken for a passing review.

Additional Context

Two things I'd flag beyond the fix:

• This is the second false-green of the same shape I've hit today (the other was in PerformanceMonitor: a green review check with no review comment). The general lesson is that "check is green" and "check did its job" are different claims, and for the review gate specifically the cheap discriminator is job duration — seconds means skipped.
• The irony worth recording: Skip the AI review job on Dependabot PRs #423 was itself merged with 0 review comments, which is exactly what this bug predicts. The change that disabled the gate was the first PR to benefit from it being disabled.

[erikdarlingdata]

Verification, since trusting the green check is what this bug punished:

Before the sync: the review runs on #426 and #427 completed green in 12-13s, with "Skipping action due to workflow validation" in the logs and no review posted.

After #429 put claude-code-review.yml on main, two runs, two trigger paths:

• Manual re-run of Give the WAM broker a parent window handle so Entra MFA works (#425) #426's review run (31520763750): zero validation warnings in the log, total_cost_usd: 0.987, Claude execution duration_ms: 203168, and it posted two substantive inline findings — an off-UI-thread Avalonia access in the handle resolver and the untested CLI refusal path. Both were real; both are fixed in 28be5ed.
• Organic pull_request trigger from the follow-up push (31541417667): again zero validation warnings, total_cost_usd: 1.219, duration_ms: 204812, full review posted, no new findings on the fixed diff.

The gate is demonstrably reviewing again on both paths. The durable rule: any edit to claude-*.yml that merges to dev needs a same-day cherry-pick PR to main, or the gate dies green until the next release. And PRs that themselves touch workflow files will keep skip-greening by design — duration stays the tell: seconds means skipped, minutes means reviewed.