[betizzel]

Summary

This runs Reposition's cache workflow on a real backlog for the first time: all open PRs of omacom/omarchy, downloaded with triage-o-mator. The test cases come from hand-checked duplicate groups that neither Reposition nor triage-o-mator produced, which covers part of roadmap steps 1 and 2.

I worked on this with Claude (Opus 5.5) as my coding agent.

What's added

• examples/omarchy-review-cases.json: 55 cases built from @aholbreich's hand-checked duplicate groups in Triage data: 78 open PRs are 16 distinct fixes (verified by shared file) omacom/omarchy#11049 (2026-09-27 update, each checked by reading the diffs). Each query is the title of the oldest open PR in a group, searched over summary, files and diff, and the expected items are the group's open PRs. The two PRs he named as the same problem solved a different way are kept as hard_negatives for later.
• validation/omarchy-backlog-trial.json: a compact receipt (16 KB) with the corpus shape, commits, index costs and one row per case.
• docs/real-backlog-trial.md: what was run, the results and the limits.
• docs/roadmap.md: a short note under steps 1 and 2 on what this covers and what is still open.

No cache or index is committed. The case file only holds PR numbers and titles, which are public.

Results

2,849 open PRs (2,643 complete, 206 with gaps), benchmarks/cache_retrieval.py with its default 12,000-byte response budget:

Other group members returned (the anchor PR itself excluded)	75 / 91 (82.4%)
Cases where every member was returned	46 / 55
Median returned items per case	5
Median warm FTS5 query (ranked, any-term)	318 ms
Median verified literal scan (cost reference only)	3,864 ms
Indexed source / index size	103.3 / 192.6 MiB
Build time / peak RSS	26.9 s / 679 MiB

• The anchor's own title always finds the anchor, so the 82.4% leaves it out
• Every case hit the 1,000-candidate limit, since title words use any-term matching
• Precision isn't measured: the labels say which PRs belong together, not how relevant everything else returned is

Testing

• python -m unittest discover -s tests passes (79 tests, 5 skipped)
• ruff check . and ruff format --check . pass
• Both JSON files load

One thing to check

AGENTS.md says to use synthetic fixtures in source control. I read that as keeping real caches and private data out, and these cases are labels over public PRs, but you know the intent better. If you'd rather keep real data out of the repo, I can move the cases and receipt to a separate place and just link them from the doc.

[Copilot]

Copilot review overview

🟡 Changes recommended

The receipt lacks content-bound corpus identifiers, and the documentation overstates label independence.

Review effort: Balanced
Findings: 1 · 2

Open (3)

• Record the corpus ID and source checkpoint in the receipt · New
• Clarify label independence and disclose candidate-generation overlap · New
• Link directly to the cited update comment · New

What changed in this PR

Adds the first real-backlog retrieval trial using independently reviewed Omarchy duplicate groups.

Changes:

• Adds 55 retrieval cases and hard negatives.
• Records retrieval, latency, and resource results.
• Documents findings and roadmap coverage.

File	Description
examples/​omarchy-review-cases.json	Defines Omarchy evaluation cases.
validation/​omarchy-backlog-trial.json	Records trial metrics and results.
docs/​real-backlog-trial.md	Documents methodology and findings.
docs/​roadmap.md	Updates roadmap progress.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jeremydixon22]

Thanks, Baytizzel. I checked the 55 cases against @aholbreich's linked update and independently reconciled the receipt totals: 130/146 known members returned, 75/91 when the query PR is excluded, and 46/55 complete groups. The corpus binding and label-provenance corrections address the earlier review notes. I'm comfortable keeping these small, public validation cases and the receipt in the repo; they are distinct from committing a user cache or private data.

One clarification before merge: the table places the 318 ms warm FTS5 query beside the 3,864 ms verified literal scan. In benchmarks/cache_retrieval.py, the scan tests a case-sensitive, whole-title substring, while FTS5 performs ranked any-term retrieval. Please state explicitly that the scan is a cost reference, not a like-for-like latency or retrieval-quality comparison. A brief machine/runtime description would also help readers interpret the timings if you have it, but that can be a follow-up.

I ran the 79 local tests and Ruff successfully. This is a useful first real-backlog baseline, with the candidate-cap and precision limits clearly stated.

[betizzel]

Thanks for checking it so carefully, Jeremy! Both are in 6f5eccd:

• Scan vs FTS5: the doc now says the literal scan is a cost reference, not a like-for-like latency or retrieval comparison. The scan looks for each case's whole title as one case-sensitive substring, while FTS5 ranks any-term matches. They really do different work: the median scan matched 2 items, while every FTS5 query hit its 1,000-candidate limit. The table rows and the receipt say the same, and I updated the PR description's table too.
• Machine: AMD Ryzen AI 7 PRO 350 (8 cores, 16 threads), 27 GiB RAM, NVMe SSD (btrfs), Linux 7.2.5, Python 3.14.7, SQLite 3.53.4. It's in the doc and in an environment block in the receipt.

Tests and Ruff still pass.

[jeremydixon22]

The requested clarifications are addressed at 6f5eccd: the verified literal scan is explicitly a cost reference with different matching semantics from ranked FTS5 retrieval, and the machine/runtime details are recorded in both the document and receipt. The 55 measured cases and evaluation labels are unchanged.

JSON and diff checks passed. Hosted CI passed all four jobs: core on Python 3.10, 3.12 and 3.14, plus optional dependencies and retrieval evaluation.