test: add live provider regression matrix

[altaywtf]

Summary

Add a committed, opt-in live regression matrix for every supported provider without making authenticated model calls part of deterministic CI.

Closes #93.

Changed

• add public synthetic clean and deliberately defective fixture overlays
• build the current CLI and review both frozen commit controls through Codex, Claude, Cursor, and Grok
• require exact commit/provider metadata, clean precision, and coverage of every planted defect file without golden prose or exact finding counts
• add mise run verify:live with provider selection and repeat controls
• repair the Grok direct live smoke after the v1.0.4 completion contract
• enforce the Grok assessment schema maximum against the untrimmed value locally

Review aids

base fixture -> clean or defective commit -> fixture go test -> current CLI -> provider -> canonical report assertions

The defective control places its three required failures in counts.go, batch.go, and config.go; extra findings such as test gaps remain valid, while missing any planted file fails the gate.

Risks

Live verification consumes session credentials, provider quota, and time. It remains outside verify, release gates, and ordinary CI. Cursor explicitly runs with web access; the other providers explicitly run with web off. Operational provider failures remain failures instead of being hidden. Repeat 10 can take more than eight hours if every review consumes its protocol retry.

Verification

• mise run verify
• repaired adapter smoke: AUTOREVIEW_TEST_LIVE_GROK=1 go test -p 1 -parallel 1 ./internal/provider -run '^TestGrokLive$' -count=1 -v
• final mise run verify:live: all 8 clean/defective controls passed in 376.49s, with valid attempt-one reports from Codex 0.147.0, Claude 2.1.233, Cursor 2026.08.11-e8db854, and Grok 1.0.4
• an earlier matrix observed one Claude provider-process failure; the focused Claude rerun and two subsequent complete matrices passed
• final Codex autoreview: clean after fixing untrimmed max-length parity, exact commit assertion, and maximum repeat timeout budgeting

Complexity

Moderate. The live orchestration is test-only Go code; runtime behavior is unchanged except for closing the Grok schema/local-validation parity gap.

[Copilot]

Pull request overview

Adds an opt-in, committed live provider regression matrix that exercises the built autoreview CLI against deterministic synthetic “clean” and “defective” commit controls, while keeping authenticated model calls out of deterministic CI. Also tightens Grok contract/schema parity by enforcing the completion assessment maximum consistently in schema + local validation.

Changes:

• Introduce a live, CLI-level provider matrix test with provider selection and bounded repeat controls, plus contributor documentation and a mise run verify:live entry point.
• Extend the public synthetic fixture with a deliberately defective overlay (planted defects across three files) and update fixture documentation.
• Fix Grok completion assessment max-length parity: schema constant + schema test + local validation + regression coverage, and update the Grok live smoke prompt.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
testdata/v0.1-fixture/README.md	Document clean vs defective fixture overlays and their review contracts.
testdata/v0.1-fixture/defective/helpers_test.go	Add nominal tests for defective overlay helpers (fixture scaffolding for planted defects).
testdata/v0.1-fixture/defective/counts.go	Add intentionally defective CountByOwner helper for defect recall coverage.
testdata/v0.1-fixture/defective/config.go	Add intentionally defective ReadConfig helper (resource management defect).
testdata/v0.1-fixture/defective/batch.go	Add intentionally defective Batch helper for correctness defect coverage.
schema/schema_test.go	Assert Grok completion assessment maxLength matches the contract constant.
schema/embed.go	Introduce GrokMaximumFileAssessmentCharacters constant and wire into Grok completion schema generation.
mise.toml	Add verify:live task to run the authenticated, opt-in provider regression matrix.
internal/provider/grok.go	Enforce Grok completion assessment maximum length during local validation.
internal/provider/grok_test.go	Add regression coverage for over-max assessment length (including whitespace).
internal/provider/grok_live_test.go	Update Grok live smoke prompt to match the v1.0.4 completion contract expectations.
CONTRIBUTING.md	Document live regression checks, provider selection, repeat bounds, and prerequisites.
cmd/autoreview/live_e2e_test.go	Add the live CLI/provider matrix e2e test and fixture materialization logic.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[slopzapper]

✨ CLEAN

The live matrix, fixture overlays, and Grok assessment cap match the documented CLI and completion contracts. No introduced defect on the eligible paths.

[slopzapper]

⚡ SLOPS ZAPPED: 1

Verdict: Zaps needed

The default live matrix cannot pin Cursor reasoning_effort and still inherits host autoreview config, so Cursor cells fail closed for a normal XDG or env setting. The fixture and Grok assessment bound look sound.

Findings

1. 🧨 MIDSLOP · P1 — Cursor live reviews inherit host reasoning_effort — broken isolation (cmd/autoreview/live_e2e_test.go:223)

[slopzapper]

✨ CLEAN

The live matrix, fixture overlay, env isolation, and Grok assessment cap match the stated contracts. No introduced defect in the eligible paths survived the diff and call-path checks.