feat: add configurable custom review types - #1091
Open
mariusvniekerk wants to merge 7 commits into
Open
Conversation
Teams need reusable review rubrics that can run through the same queue, panel, CI, severity, and verdict workflows as built-in reviews. Fixed prompts could not support project-specific audits such as a thermonuclear quality review without changing roborev itself. Custom types use local templates and native schema-constrained agent output. Roborev keeps control of diff context, filtering, canonical review text, and the stored pass or fail verdict. Local-only file resolution also avoids network-dependent reviews, while CI reads repository templates from the trusted base ref. Generated with Codex Co-authored-by: Codex <noreply@openai.com>
Custom review jobs could run with a changed or untrusted rubric. Daemon-free CI read repository config and relative template files from the review checkout, while queued jobs silently used a generic prompt if their type disappeared. Resolve CI definitions from the pre-review commit, honor each custom type's reasoning, and reject rubrics that leave less than 16 KiB for review context. This keeps verdicts tied to the intended policy and the code under review. Generated with Codex Co-authored-by: Codex <noreply@openai.com>
mariusvniekerk
force-pushed
the
t3code/custom-review-types
branch
from
August 22, 2026 22:20
bc1e5d4 to
212d39c
Compare
This comment has been minimized.
This comment has been minimized.
Custom review types can require a specific structured-output agent and model. Later workflow checks treated those settings differently from initial config resolution, so a review could silently use another available agent or model. Use the complete custom type definition at each workflow resolution step. Repository definitions continue to replace same-name global definitions. Generated with Codex Co-authored-by: Codex <noreply@openai.com>
This comment has been minimized.
This comment has been minimized.
The schema-constrained JSON is the source of truth for a custom review verdict. The worker built the prompt with the effective severity threshold but filtered the JSON with only the job-level value, so inherited thresholds could produce the wrong pass or fail result. Carry the resolved threshold through structured result processing so prompt guidance and verdict calculation use the same policy. Generated with Codex Co-authored-by: Codex <noreply@openai.com>
Schema-constrained JSON is the source of truth for custom review verdicts. Automatic closing and completion events reparsed the rendered Markdown, so their result could disagree with the verdict stored from the JSON findings. Resolve the verdict once and pass it to every downstream consumer. Prose and synthesis reviews continue to use the existing deterministic parser. Generated with Codex Co-authored-by: Codex <noreply@openai.com>
This comment has been minimized.
This comment has been minimized.
Custom review type validation can load repository configuration, so validation tests must run in a real repository. The test previously relied on the source checkout's Git metadata, which is absent from Nix source archives. Use a temporary repository for every validation case so packaging checks test the command contract instead of the build environment. Generated with Codex Co-authored-by: Codex <noreply@openai.com>
This comment has been minimized.
This comment has been minimized.
Schema-constrained JSON is the source of truth for custom review verdicts. Daemon-free batches and panel synthesis dropped that result and reparsed the rendered Markdown, so a summary could reverse a severity-filtered verdict. Carry the explicit verdict with review results and stored panel members, and preserve it when a panel passes one member through. Prose reviews retain their existing parser. Reject custom names that belong to built-in workflows so their agent and model settings cannot be silently replaced. Review triage intentionally left user-selected external template paths and the existing prompt-size configuration source unchanged. Those reports did not identify a missing size limit or a capability that the user lacked before this change. Sources: #1091 (comment) #1091 (comment) Generated with Codex Co-authored-by: Codex <noreply@openai.com>
roborev: Combined Review (
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roborev's built-in review types cover common review workflows, but they do not
provide a way to package a repository-specific rubric as a named review type.
This adds configurable custom types backed by Go templates, with named file
includes and optional per-type agent, model, and reasoning overrides.
Custom reviews use a Roborev-owned JSON Schema for the summary and findings.
Roborev validates severity values, applies the configured minimum severity,
derives the pass/fail verdict, and renders the final review text. Built-in
review types keep their existing prose path.
Repository-defined types are treated as trusted configuration. Daemon-free CI
loads their configuration, templates, and repository-relative includes from
the trusted side of the reviewed range; runtime URL fetching is intentionally
unsupported. Queued jobs fail explicitly if their type disappears, and prompt
construction reserves 16 KiB for the review context so a large rubric cannot
crowd out the code under review.
The documentation covers configuration precedence, path rules, template
values, compatible structured-output agents, severity guidance, and a complete
thermonuclearexample based on the Cursor Team Kit rubric.