[soheimam]

What changed? Why?

Step 2 of the reviewer fixes from the BKB feedback: a decision model for the calls the reviewer gets wrong most often. Stacked on #2166, so this diff shows step 2 only. It will retarget to master when #2166 merges.

New scripts/docs-pr-review/decisions.mjs calls the LLM Gateway Decisions API (POST /v1/decisions, gpt-6-luna). It's a classifier that returns probabilities, at about 300 ms and about $0.0001 per call. It makes two checks:

• Findings that would not mislead a reader. Each non-blocking finding is scored with one yes/no question: would the text it's about give a reader a wrong or unsupported fact, a wrong step, or a claim that could cause a mistake? Below 0.5, the finding stays in the summary under a collapsed "Minor suggestions" list and is not posted inline. Nothing is deleted. Blocking findings are never scored. This targets the style nits reviewers rejected on low-risk PRs.
• Values that could cost funds. Each PR is scored on whether a wrong value in the added lines (address, endpoint, chain ID, fee, amount, key handling) could cost readers funds. At 0.8 or above the level is raised to high, with the reason and a "What to check" line. It can only raise the level, never lower it, so untrusted PR text can't talk a PR down.

Both checks fail open. A missing key, HTTP error, refusal, bad answer or 15 s timeout logs a warning, and the review posts as it does today. The run notice now includes minor= and funds_risk=.

Notes to reviewers

• Evidence. Thresholds were set on 18 findings with a clear human verdict (soheimam on the BKB PRs, roethke on docs(base-chain): add fast bridges, aggregators, and messaging protocols to Bridge to Base #2159), 9 accepted and 9 rejected:
  • At 0.5, "misleads" agreed with the human 14 of 18 times.
  • It would have moved 7 of the 9 rejected findings out of inline comments and kept 7 of the 9 accepted ones inline.
  • Misses: docs(proofs): document per-game proposal interval selection #2122 (placement) and one docs(base-chain): add fast bridges, aggregators, and messaging protocols to Bridge to Base #2159 link fix were accepted but scored low; docs(validity): document predicate limit and unspecified behaviors #2115 and docs(base-verify): add base mainnet identity attestation options #2141 were rejected but scored high.
• Funds risk on the 27 BKB PRs. It scored 0.98 on docs(bridges): document native usdc on base vs bridged usdbc #2132 (3 new token addresses, already high), 0.51 on docs(validity): document b20 and policy registry storage slots #2139, and under 0.07 on the rest. It changes no current label and works as a safety net for values the regex rules miss.
• 18 labels is a small sample. The thresholds are constants in decisions.mjs, to re-check against the eval set.
• Host and access. The endpoint is on the gateway host the workflow already allows (llm-gateway.coinbase-corp.com), so no egress change. I verified access with a personal gateway key. The Action's LLM_GATEWAY_API_KEY is first exercised on this PR's next review run; if it lacks access, the checks skip with a warning.
• Conformance with the OpenAI Decisions guide:
  • Same endpoint, Bearer auth, and {model, input, questions} body as its curl examples. Predicate questions carry type, name and instructions, and answers are read by name and probability.
  • Refused questions are handled per answer: a refused or failed finding stays as it is.
  • Findings are scored in separate requests because each has its own input; the guide puts questions in one request only when they share input.
  • Thresholds come from labeled examples, as the guide advises.
  • The guide writes predicate instructions as questions. I tried that wording on the same 18 findings and it did slightly worse (13 of 18 against 14), so the current wording stays.
• Contract addresses pass through the gateway's secret redaction unchanged (checked).

How has it been tested?

• npm test: 204 pass. New decisions.test.mjs covers the request shape, error, refusal and timeout handling, failing open, hunk selection, threshold logic, raiseRisk, and the collapsed list.
• Ran the module against the live gateway on the 18 labeled findings and on docs(bridges): document native usdc on base vs bridged usdbc #2132, docs(security): fix bug bounty page descriptions for search #2135, docs(rpc): add batch balance snapshot guidance to eth_getBalance #2138 and docs(validity): document b20 and policy registry storage slots #2139.

Screenshots

N/A (no user-facing changes)

Generated with Toshi

[github-actions]

Docs review

Risk: Medium (one AI review with a larger model) · risk: medium · shadow mode, not enforced

• changes code in scripts/docs-pr-review/__tests__/decisions.test.mjs
• changes code in scripts/docs-pr-review/decisions.mjs
• changes code in scripts/docs-pr-review/index.mjs
• changes code in scripts/docs-pr-review/render.mjs
• changes code in scripts/docs-pr-review/risk.mjs
• changes 396 lines across 5 files, not counting generated index files (low allows up to 20 lines and 3 files)

What to check:

• Run the changed code or check its output.
• Large changes are easier to get wrong. Check the facts in every changed section.

Lower-level signals

• Low: repeats a contract address already in the docs (not a risk signal)

No documentation pages changed in 2f1e433, so there is no content review.

Advisory review from the Docs PR review workflow (run). It does not approve, block, or edit this PR. Add the skip-ai-review label to turn it off for this PR.

[cb-heimdall]

✅ Heimdall Review Status

Requirement	Status	More Info
Reviews	✅ 1/1	Denominator calculation Show calculation 1 if user is bot 0 1 if user is external 0 2 if repo is sensitive 0 From .codeflow.yml 1 Additional review requirements Show calculation Max 0 0 From CODEOWNERS 0 Global minimum 0 Max 1 1 1 if commit is unverified 0 Sum 1