Provide commit-by-commit LLM reviews?

[maflcko]

This is mostly for brainstorming, because I don't know if it will be useful and provide more value than it consumes.

In theory, I guess it could be implemented by checking out each commit and running something like codex -c model=gpt-5.1-codex-mini --dangerously-bypass-approvals-and-sandbox exec 'review the top commit. Only mention high severity issues. Use markdown. If no issues found, reply with exactly "No severe review issues found".' > /tmp/result_for_commit.txt.

However, I'll need to figure out how to supply such an agent-cli inside docker, clone and checkout all pull request commits, and write the handling code. Not sure if this is worth it.

[katesalazar]

If features are to be reviewed commit-by-commit, intuitively seems useful LLM to be eventually applied commit-by-commit too.

[maflcko]

Looks like some devs use claude (https://delvingbitcoin.org/t/using-ai-tooling-for-code-review/2277), but more in an interactive way, so I am not sure if and how this could be integrated non-interactively.

[maflcko]

Looks like some projects are using elaborate tooling (https://github.com/facebook/rocksdb/blob/main/claude_md/ci_review_prompt.md), but I don't know how applicable this is to Bitcoin Core, and how useful/effective this is overall.

[maflcko]

There was a recent discussion on IRC yesterday:

...
[18:39] <andrewtoth> l0rinc: I don't want automatic comments from bots. If a reviewer uses a smart AI and then triages the results in their own words, ensuring no false positives, that would be welcome.
[18:39] <tigerMafia> andrewtoth: +1
[18:40] <lightlike> agree, AI during review is most helpful if you can interact with it / steer it into directions where real issues may be. an automatic AI review isn't helpful, everyone can do a rough analysis locally and then take it from there.
[18:40] <andrewtoth> I can run my own AI reviewer myself, I don't need it posting on my PR
[18:40] <l0rinc> andrewtoth: I sympathise with the feeling, but some PRs have so many bugs that I would prefer an AI point them out so by the time we get to it the trivial ones are already handled
[18:41] <andrewtoth> no
[18:41] <l0rinc> yes :)
[18:41] <eugenesiegel> are there certain areas that LLMs are lacking in or are security engineers cooked?
[18:41] <achow101> i absolutely do not want automated review
[18:42] <maxedw> also not a fan of automated review, I use privately and find 4/5 complaints are non-issues or random opinion
...

So I wonder if this will be used at all?

I think it is clear that nothing should be posed as GitHub comments for now. Though, if this is just an external static html with the review, no one will read it?

[maflcko]

Maybe someone could try to run it on the current most recent 30 pull requests with:

for pull in most_recent(n=30):
 review > f"out_{pull.id}.txt"

and then check if the signal-to-noise ratio is all right (I'd say it must have at least 4/5 comments that are relevant and useful)