diff --git a/docs/real-backlog-trial.md b/docs/real-backlog-trial.md new file mode 100644 index 0000000..4f381ca --- /dev/null +++ b/docs/real-backlog-trial.md @@ -0,0 +1,71 @@ +# First real-backlog trial: omacom/omarchy + +This is the first run of the immutable-cache workflow against a real backlog +instead of a synthetic fixture. It covers roadmap steps 1 and 2 for one +repository, one machine and one run; it does not qualify the defaults. + +## Source and labels + +triage-o-mator's bulk `backlog` acquisition froze all 2,849 open PRs of +omacom/omarchy on 2026-09-30: 2,643 complete and 206 with recorded gaps. No +cache is committed here; the [receipt](../validation/omarchy-backlog-trial.json) +records the corpus ID, its progress checkpoint, the inventory snapshot and the +base commit, so a rerun can tell whether it reads the same input. + +The labels are independent of Reposition and triage-o-mator. In an +[update to omacom/omarchy#11049](https://github.com/omacom/omarchy/issues/11049#issuecomment-5857142203), +@aholbreich published 28 duplicate groups and 27 pairs of open PRs on +2026-09-27. He found candidates by clustering titles and requiring a shared +primary file, then checked each one by reading the diffs. That candidate step +overlaps with searching summaries and file lists, so the cases may favour +duplicates that lexical retrieval finds easily; the manual review settles +membership, not how the candidates were found. The +[case file](../examples/omarchy-review-cases.json) keeps the 55 groups with two +or more open PRs. Each query is the oldest open member's own title, searched over +`summary`, `files` and `diff`: where a triager asking "is #N a duplicate?" +starts. That query choice is ours, not the labeller's. The two PRs the labeller +named as the same problem solved differently are kept as `hard_negatives` for a +later decision-level evaluation; retrieval returning them is expected. + +## Results + +| Measure | Value | +| --- | ---: | +| Expected items returned within 12,000 bytes | 130 / 146 | +| Other group members returned (anchor excluded) | 75 / 91 (82.4%) | +| Cases with every member returned | 46 / 55 | +| Median returned items per case | 5 | +| Median warm FTS5 query (ranked, any-term) | 318 ms | +| Median verified literal scan (cost reference only) | 3,864 ms | +| Indexed source / index size | 103.3 / 192.6 MiB | +| Build time / peak RSS | 26.9 s / 679 MiB | + +The anchor's own title always finds the anchor, so the 82.4% figure is the one +that measures finding its duplicates. Every case hit the 1,000-candidate limit: +a title's ordinary terms use any-term matching, so ranking, not filtering, does +the work. The index is 1.86 times the indexed source. + +The literal scan is a cost reference, not a like-for-like latency or retrieval +comparison. The benchmark's scan looks for each case's whole title as one +case-sensitive substring in every verified projection, while the FTS5 query +ranks any-term matches. They answer different questions: the median scan +matched 2 items, while every FTS5 query reached its 1,000-candidate limit. The +scan time shows what reading and verifying every projection costs, not how much +faster ranked search finds the same results. + +Measured on an AMD Ryzen AI 7 PRO 350 (8 cores, 16 threads) with 27 GiB of +RAM and an NVMe SSD (btrfs), Linux 7.2.5, Python 3.14.7 and SQLite 3.53.4. + +## Limits + +Precision is not measured: the labels say which PRs belong together, not how +relevant each other returned item is. A returned item is a lead to read, never +a duplicate decision. Timings are from one run on one machine with OS caches not +cleared and nothing else running. Recreating the cache needs GitHub access and a +triage-o-mator install, and a new acquisition gets a new corpus ID and +checkpoint; compare them with the receipt. Rerun with: + +```sh +python benchmarks/cache_retrieval.py --cache /path/to/install/data/omacom/omarchy/cache \ + --corpus CORPUS_ID --cases examples/omarchy-review-cases.json +``` diff --git a/docs/roadmap.md b/docs/roadmap.md index 7e257f7..eab3a0e 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -12,8 +12,12 @@ exercise their contracts; they do not qualify production-scale performance. 1. Trial the prepared integration against representative real caches and the existing triage TUI. Measure disk overhead, peak memory, selective/dense query latency, omissions and independent review effort before widening the defaults. + The [first real-backlog trial](real-backlog-trial.md) covers one omacom/omarchy + corpus on one machine; review effort and other repositories remain open. 2. Obtain independently reviewed relevance and hard-negative cases. Measure - review decisions and effort as well as retrieval quality. + review decisions and effort as well as retrieval quality. That trial uses 55 + independently hand-checked duplicate groups; graded relevance, measured + precision and review decisions are still missing. 3. Qualify optional lexical/semantic fusion and anchored evidence packing against the default with identical source pools and final consumer budgets. 4. Add source acquisition or incremental refresh only when the calling workflows diff --git a/examples/omarchy-review-cases.json b/examples/omarchy-review-cases.json new file mode 100644 index 0000000..5a6f68f --- /dev/null +++ b/examples/omarchy-review-cases.json @@ -0,0 +1,837 @@ +{ + "label_authority": "Hand-checked duplicate groups for omacom/omarchy by @aholbreich (2026-09-27 update to omacom/omarchy#11049). He found candidates by clustering titles and requiring a shared primary file, then checked each by reading the diffs, so the labels are independent of Reposition and triage-o-mator but share a lexical candidate step with this retrieval. Queries are the oldest open member's title, chosen by us, not by the labeller. Membership is limited to PRs open in the ledger used to build the cases.", + "source": "https://github.com/omacom/omarchy/issues/11049#issuecomment-5857142203", + "hard_negatives": [ + { + "pr": 10113, + "near": [ + 9605, + 10411, + 11994, + 12019, + 12264, + 12323, + 13154, + 13213 + ], + "why": "fixes the same windows-vm bug the other way round, by accepting the mode instead of clearing it" + }, + { + "pr": 11803, + "near": [ + 7023, + 12022, + 12150 + ], + "why": "documents the current toggle bar behaviour instead of flipping it" + } + ], + "cases": [ + { + "id": "omarchy-9605-omarchy-windows-vm-clear-setgid-on-mount", + "query": "Clear set-ID bits when securing the Windows VM mount sources", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:9605", + "pr:10411", + "pr:12019", + "pr:12264", + "pr:12323", + "pr:13154", + "pr:13213" + ] + }, + { + "id": "omarchy-7074-remember-menu-position-when-navigating-b", + "query": "Remember cursor position when navigating back in the root menu", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7074", + "pr:7954", + "pr:11577", + "pr:12098", + "pr:12644", + "pr:12759", + "pr:13012", + "pr:13225" + ] + }, + { + "id": "omarchy-8879-enable-sunshine-by-its-real-unit-name", + "query": "Use Sunshine's packaged systemd unit", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8879", + "pr:12271", + "pr:12358", + "pr:12472", + "pr:12505", + "pr:13230", + "pr:13331" + ] + }, + { + "id": "omarchy-7186-battery-status-across-all-packs", + "query": "Report combined dual-battery status in the power panel", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7186", + "pr:10392", + "pr:12024", + "pr:12445" + ] + }, + { + "id": "omarchy-7023-omarchy-toggle-bar-on-off-inverted", + "query": "Fix inverted on/off semantics in omarchy-toggle-bar", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7023", + "pr:12022", + "pr:12150" + ] + }, + { + "id": "omarchy-6485-grok-usage-collector", + "query": "Add a Grok usage collector", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:6485", + "pr:12352", + "pr:13219" + ] + }, + { + "id": "omarchy-5774-vivaldi-browser-support", + "query": "Add Vivaldi browser support (fixed)", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:5774", + "pr:9430", + "pr:12754" + ] + }, + { + "id": "omarchy-7283-clipboard-shortcuts-on-non-latin-layouts", + "query": "Use layout-independent universal clipboard shortcuts", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7283", + "pr:9130", + "pr:13007" + ] + }, + { + "id": "omarchy-6647-hermes-usage-collector", + "query": "Add local and remote Hermes usage sources to Agents panel", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:6647", + "pr:9871" + ] + }, + { + "id": "omarchy-7016-fuzzy-menu-search", + "query": "Added fuzzy search to menu", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7016", + "pr:8987" + ] + }, + { + "id": "omarchy-8065-opencode-go-usage-collector", + "query": "Add OpenCode Go usage collector for the agents panel", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8065", + "pr:12651" + ] + }, + { + "id": "omarchy-8479-antigravity-usage-collector-agy-vs-antig", + "query": "Add Antigravity (agy) agent usage collector and test suite", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8479", + "pr:11989" + ] + }, + { + "id": "omarchy-11415-dismiss-low-battery-warning-on-charge", + "query": "Dismiss stale low battery warnings", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11415", + "pr:11959" + ] + }, + { + "id": "omarchy-10476-calendar-grid-shifts-on-midnight-dst-swi", + "query": "Build the calendar month grid in UTC", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:10476", + "pr:13096", + "pr:13127", + "pr:13186" + ] + }, + { + "id": "omarchy-8019-show-hide-toggle-for-the-wi-fi-passphras", + "query": "Eye toggle to show/hide the Wi-Fi password", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8019", + "pr:11720", + "pr:12891", + "pr:12925" + ] + }, + { + "id": "omarchy-10364-keyboard-backlight-lost-after-repeated-b", + "query": "Preserve keyboard backlight across repeated blanks", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:10364", + "pr:10366", + "pr:11404", + "pr:12960" + ] + }, + { + "id": "omarchy-7592-lock-screen-password-focus-after-resume", + "query": "Refocus the lock password field after resume", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7592", + "pr:8560", + "pr:12435" + ] + }, + { + "id": "omarchy-9168-kitty-bold-italic-faces-on-font-set", + "query": "fix(font): reset explicit kitty bold/italic font variants to auto on font set", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:9168", + "pr:12171", + "pr:13235" + ] + }, + { + "id": "omarchy-10194-display-panel-toggle-under-the-lua-confi", + "query": "Fix the display panel's on/off rows under the Lua config parser", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:10194", + "pr:11112", + "pr:13223" + ] + }, + { + "id": "omarchy-7723-keep-the-last-valid-shell-json", + "query": "Keep last good shell config when user shell.json fails to parse", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7723", + "pr:12992", + "pr:13104" + ] + }, + { + "id": "omarchy-12935-fontconfig-monospace-rule-as-a-conf-d-dr", + "query": "Write fontconfig monospace override to conf.d drop-in", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:12935", + "pr:12999", + "pr:13092" + ] + }, + { + "id": "omarchy-7564-keybindings-menu-lua-bind-scan", + "query": "Make the keybindings menu's Lua bind scan safe against load-time config reads", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7564", + "pr:12916", + "pr:13276" + ] + }, + { + "id": "omarchy-7661-rank-apps-above-menu-actions-in-search", + "query": "fix(menu): rank apps above destructive actions", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7661", + "pr:11730", + "pr:12223" + ] + }, + { + "id": "omarchy-8084-ssh-reconnect-leaving-the-alternate-scre", + "query": "Leave the alternate screen only when the connection dropped", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8084", + "pr:12063", + "pr:13116" + ] + }, + { + "id": "omarchy-7042-ctrl-n-ctrl-p-menu-navigation", + "query": "Add Ctrl-N and Ctrl-P menu navigation", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7042", + "pr:10934" + ] + }, + { + "id": "omarchy-7680-lock-screen-password-reveal-toggle", + "query": "Add a reveal toggle to the lock screen password field", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7680", + "pr:13183" + ] + }, + { + "id": "omarchy-8824-orphan-prompt-under-omarchy-update-y", + "query": "Skip orphan prompt during unattended updates", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8824", + "pr:13371" + ] + }, + { + "id": "omarchy-8590-unset-omarchy-path-in-update-dev", + "query": "Handle unset OMARCHY_PATH in update-dev", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8590", + "pr:12860" + ] + }, + { + "id": "omarchy-7783-same-lines-same-fix", + "query": "Keep PwNode objects out of the audio panel's Repeater models", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7783", + "pr:12658" + ] + }, + { + "id": "omarchy-11069-same-lines-same-fix", + "query": "Run declared plugin cleanup before removal", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11069", + "pr:11470" + ] + }, + { + "id": "omarchy-8429-same-lines-same-fix", + "query": "[Security] Keep the update transcript out of world-writable /tmp", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8429", + "pr:12957" + ] + }, + { + "id": "omarchy-10413-same-lines-same-fix", + "query": "Harden omarchy-refresh-config against path traversal", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:10413", + "pr:13214" + ] + }, + { + "id": "omarchy-13041-same-lines-same-fix", + "query": "Honour explicit expireTimeout for critical notifications (#12911)", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:13041", + "pr:13100" + ] + }, + { + "id": "omarchy-13044-same-lines-same-fix", + "query": "fix(launch-editor): pass wait flags to GUI editors in inline mode (#13037)", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:13044", + "pr:13094" + ] + }, + { + "id": "omarchy-8875-same-lines-same-fix", + "query": "Install lib32 GPU drivers before Steam", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8875", + "pr:12569" + ] + }, + { + "id": "omarchy-11235-same-lines-same-fix", + "query": "Stop speedtest workers when their parent is killed", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11235", + "pr:12196" + ] + }, + { + "id": "omarchy-11344-same-lines-same-fix", + "query": "Support both zed and zeditor editor binaries", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11344", + "pr:13236" + ] + }, + { + "id": "omarchy-11364-same-lines-same-fix", + "query": "Keep existing wine when installing Lutris", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11364", + "pr:12089" + ] + }, + { + "id": "omarchy-11893-same-lines-same-fix", + "query": "fix(screenrecording): escalate webcam cleanup to SIGKILL and clear on stop-recording", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11893", + "pr:12021" + ] + }, + { + "id": "omarchy-11918-same-lines-same-fix", + "query": "Back off fingerprint retries that fail immediately", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11918", + "pr:12544" + ] + }, + { + "id": "omarchy-11273-same-lines-same-fix", + "query": "Lock screen: keep fingerprint idle while the display is blanked", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11273", + "pr:12420" + ] + }, + { + "id": "omarchy-12585-same-lines-same-fix", + "query": "Keep the Bluetooth widget on the bar while the radio is off", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:12585", + "pr:13204" + ] + }, + { + "id": "omarchy-10749-same-lines-same-fix", + "query": "Keep the Bluetooth bar icon visible when there's no adapter", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:10749", + "pr:12241" + ] + }, + { + "id": "omarchy-12649-same-lines-same-fix", + "query": "Keep UPower rate when sysfs power read fails with ENODEV", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:12649", + "pr:12755" + ] + }, + { + "id": "omarchy-11565-same-lines-same-fix", + "query": "Scope LocalSend firewall rules to private and local subnets (#11560)", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11565", + "pr:12831" + ] + }, + { + "id": "omarchy-11933-same-lines-same-fix", + "query": "Install Bitwarden desktop even when CLI conflicts with nodejs", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11933", + "pr:12870" + ] + }, + { + "id": "omarchy-10159-same-lines-same-fix", + "query": "Match Google Meet PiP titles that use an en dash", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:10159", + "pr:11169" + ] + }, + { + "id": "omarchy-9847-same-lines-same-fix", + "query": "Keep Quickshell logs from exhausting the user runtime dir", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:9847", + "pr:13314" + ] + }, + { + "id": "omarchy-8692-same-lines-same-fix", + "query": "Resolve mise wrapper binaries before execution", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8692", + "pr:13090" + ] + }, + { + "id": "omarchy-8414-same-lines-same-fix", + "query": "Detect Microarray MAFP fingerprint reader", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8414", + "pr:12936" + ] + }, + { + "id": "omarchy-10877-same-lines-same-fix", + "query": "Target the actual NVMe controller in Apple suspend fix", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:10877", + "pr:11624" + ] + }, + { + "id": "omarchy-8406-same-lines-same-fix", + "query": "Keep system binaries ahead of the mise shims in the uwsm session", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8406", + "pr:13364" + ] + }, + { + "id": "omarchy-11728-same-lines-same-fix", + "query": "fix(browser): support wrapped Exec commands in browser and webapp launchers", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:11728", + "pr:12533" + ] + }, + { + "id": "omarchy-8023-same-lines-same-fix", + "query": "Accept numpad and AltGr input in the menu and image picker filters", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:8023", + "pr:12574" + ] + }, + { + "id": "omarchy-7833-same-lines-same-fix", + "query": "Remove the toast when a sender closes its own notification", + "components": [ + "summary", + "files", + "diff" + ], + "max_snippets_per_item": 1, + "expected_items": [ + "pr:7833", + "pr:11338" + ] + } + ] +} diff --git a/validation/omarchy-backlog-trial.json b/validation/omarchy-backlog-trial.json new file mode 100644 index 0000000..8cbdf3d --- /dev/null +++ b/validation/omarchy-backlog-trial.json @@ -0,0 +1,695 @@ +{ + "schema": "reposition.real-backlog-trial.v1", + "measured_at": "2026-09-30", + "environment": { + "cpu": "AMD Ryzen AI 7 PRO 350 (8 cores, 16 threads)", + "memory_gib": 27, + "storage": "NVMe SSD, btrfs", + "os": "Linux 7.2.5", + "python": "3.14.7", + "sqlite": "3.53.4" + }, + "repository": "omacom/omarchy", + "reposition_commit": "7a76620821a4ddf0423d89e98f908700ea7aae94", + "acquisition": { + "tool": "triage-o-mator bin/cache corpus-run --bulk, profile backlog, scope open-prs", + "triage_o_mator_base_commit": "d329d6a4a738900e7dec15930635efe8298b6c47", + "corpus_id": "4c02b4b0e3555733e87406cf72edb88e23c493f1e02bfa3a93d5b8b9c03285d1", + "corpus_checkpoint": "ff0bd6495c6a3034681d7456f904a25c9ce03fa60f651cd41b24b60afe78aa8b", + "corpus_state_updated_at": "2026-09-30T18:05:20.939091+00:00", + "inventory_snapshot": "350e61a6289c21fae00685c983af5983fb5235acffa283b3378f238500244507", + "checkpoint_note": "bin/cache corpus-list checkpoint of the finished corpus; its state was last updated before the trial ran and not after", + "inventory_items": 4971, + "corpus_members": 2849, + "complete": 2643, + "with_gaps": 206 + }, + "cases_file": "examples/omarchy-review-cases.json", + "label_authority": "Hand-checked duplicate groups for omacom/omarchy by @aholbreich (2026-09-27 update to omacom/omarchy#11049). He found candidates by clustering titles and requiring a shared primary file, then checked each by reading the diffs, so the labels are independent of Reposition and triage-o-mator but share a lexical candidate step with this retrieval. Queries are the oldest open member's title, chosen by us, not by the labeller. Membership is limited to PRs open in the ledger used to build the cases.", + "components": [ + "summary", + "files", + "diff" + ], + "response_max_bytes": 12000, + "index": { + "source_cache_bytes": 141961496, + "indexed_source_object_bytes": 108320045, + "index_bytes": 201940992, + "units": 101181, + "objects": 11042, + "build_seconds": 26.918490072999703, + "peak_process_rss_bytes": 711888896 + }, + "summary": { + "cases": 55, + "expected_items_returned": [ + 130, + 146 + ], + "other_members_returned": [ + 75, + 91 + ], + "cases_with_full_recall": 46, + "median_returned_items": 5, + "median_warm_query_ms": 318.5, + "median_literal_scan_ms": 3863.5, + "cases_with_truncated_candidates": 55, + "literal_scan_note": "Cost reference, not a like-for-like comparison: the scan matches each case's whole title as one case-sensitive substring in every verified projection, while the FTS5 query ranks any-term matches.", + "median_literal_matched_items": 2 + }, + "limits": "One machine, one run, OS caches not cleared, nothing else running. The query is the oldest open member's own title, so that member is always found; other_members_returned excludes it. Labels are hand-checked same-fix groups, not relevance grades for every returned item, so precision is not measured. Retrieval is not a duplicate decision.", + "cases": [ + { + "id": "omarchy-9605-omarchy-windows-vm-clear-setgid-on-mount", + "expected": 7, + "returned": 5, + "missed": [ + "pr:10411", + "pr:12264", + "pr:12323", + "pr:13154" + ], + "known_positive_recall": 0.429, + "warm_median_ms": 356.7, + "literal_scan_ms": 3862.9, + "output_bytes": 10401, + "candidates_truncated": true + }, + { + "id": "omarchy-7074-remember-menu-position-when-navigating-b", + "expected": 8, + "returned": 5, + "missed": [ + "pr:12098", + "pr:12644", + "pr:13012" + ], + "known_positive_recall": 0.625, + "warm_median_ms": 410.9, + "literal_scan_ms": 3811.2, + "output_bytes": 10867, + "candidates_truncated": true + }, + { + "id": "omarchy-8879-enable-sunshine-by-its-real-unit-name", + "expected": 7, + "returned": 5, + "missed": [ + "pr:12358", + "pr:13230" + ], + "known_positive_recall": 0.714, + "warm_median_ms": 251.3, + "literal_scan_ms": 3881.8, + "output_bytes": 11774, + "candidates_truncated": true + }, + { + "id": "omarchy-7186-battery-status-across-all-packs", + "expected": 4, + "returned": 5, + "missed": [ + "pr:12024" + ], + "known_positive_recall": 0.75, + "warm_median_ms": 280.4, + "literal_scan_ms": 3855.5, + "output_bytes": 11842, + "candidates_truncated": true + }, + { + "id": "omarchy-7023-omarchy-toggle-bar-on-off-inverted", + "expected": 3, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 393.6, + "literal_scan_ms": 3863.8, + "output_bytes": 11370, + "candidates_truncated": true + }, + { + "id": "omarchy-6485-grok-usage-collector", + "expected": 3, + "returned": 6, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 223.5, + "literal_scan_ms": 3802.6, + "output_bytes": 11883, + "candidates_truncated": true + }, + { + "id": "omarchy-5774-vivaldi-browser-support", + "expected": 3, + "returned": 6, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 227.9, + "literal_scan_ms": 3811.4, + "output_bytes": 11920, + "candidates_truncated": true + }, + { + "id": "omarchy-7283-clipboard-shortcuts-on-non-latin-layouts", + "expected": 3, + "returned": 4, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 241.4, + "literal_scan_ms": 3833.8, + "output_bytes": 10136, + "candidates_truncated": true + }, + { + "id": "omarchy-6647-hermes-usage-collector", + "expected": 2, + "returned": 6, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 313.6, + "literal_scan_ms": 3780.8, + "output_bytes": 11985, + "candidates_truncated": true + }, + { + "id": "omarchy-7016-fuzzy-menu-search", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 227.9, + "literal_scan_ms": 3832.4, + "output_bytes": 10312, + "candidates_truncated": true + }, + { + "id": "omarchy-8065-opencode-go-usage-collector", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 281.7, + "literal_scan_ms": 3845.5, + "output_bytes": 11286, + "candidates_truncated": true + }, + { + "id": "omarchy-8479-antigravity-usage-collector-agy-vs-antig", + "expected": 2, + "returned": 4, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 332.8, + "literal_scan_ms": 3809.9, + "output_bytes": 10239, + "candidates_truncated": true + }, + { + "id": "omarchy-11415-dismiss-low-battery-warning-on-charge", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 199.0, + "literal_scan_ms": 3829.7, + "output_bytes": 11768, + "candidates_truncated": true + }, + { + "id": "omarchy-10476-calendar-grid-shifts-on-midnight-dst-swi", + "expected": 4, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 219.9, + "literal_scan_ms": 3807.9, + "output_bytes": 10756, + "candidates_truncated": true + }, + { + "id": "omarchy-8019-show-hide-toggle-for-the-wi-fi-passphras", + "expected": 4, + "returned": 5, + "missed": [ + "pr:12925" + ], + "known_positive_recall": 0.75, + "warm_median_ms": 305.0, + "literal_scan_ms": 3803.3, + "output_bytes": 11241, + "candidates_truncated": true + }, + { + "id": "omarchy-10364-keyboard-backlight-lost-after-repeated-b", + "expected": 4, + "returned": 5, + "missed": [ + "pr:10366" + ], + "known_positive_recall": 0.75, + "warm_median_ms": 232.5, + "literal_scan_ms": 3811.7, + "output_bytes": 10400, + "candidates_truncated": true + }, + { + "id": "omarchy-7592-lock-screen-password-focus-after-resume", + "expected": 3, + "returned": 4, + "missed": [ + "pr:12435", + "pr:8560" + ], + "known_positive_recall": 0.333, + "warm_median_ms": 290.4, + "literal_scan_ms": 3854.3, + "output_bytes": 10039, + "candidates_truncated": true + }, + { + "id": "omarchy-9168-kitty-bold-italic-faces-on-font-set", + "expected": 3, + "returned": 4, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 397.4, + "literal_scan_ms": 3809.4, + "output_bytes": 9846, + "candidates_truncated": true + }, + { + "id": "omarchy-10194-display-panel-toggle-under-the-lua-confi", + "expected": 3, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 403.2, + "literal_scan_ms": 3823.1, + "output_bytes": 11385, + "candidates_truncated": true + }, + { + "id": "omarchy-7723-keep-the-last-valid-shell-json", + "expected": 3, + "returned": 4, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 529.7, + "literal_scan_ms": 3823.7, + "output_bytes": 10137, + "candidates_truncated": true + }, + { + "id": "omarchy-12935-fontconfig-monospace-rule-as-a-conf-d-dr", + "expected": 3, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 289.1, + "literal_scan_ms": 3838.2, + "output_bytes": 11358, + "candidates_truncated": true + }, + { + "id": "omarchy-7564-keybindings-menu-lua-bind-scan", + "expected": 3, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 434.0, + "literal_scan_ms": 3826.7, + "output_bytes": 11416, + "candidates_truncated": true + }, + { + "id": "omarchy-7661-rank-apps-above-menu-actions-in-search", + "expected": 3, + "returned": 6, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 290.5, + "literal_scan_ms": 3818.3, + "output_bytes": 11988, + "candidates_truncated": true + }, + { + "id": "omarchy-8084-ssh-reconnect-leaving-the-alternate-scre", + "expected": 3, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 497.6, + "literal_scan_ms": 3847.1, + "output_bytes": 11403, + "candidates_truncated": true + }, + { + "id": "omarchy-7042-ctrl-n-ctrl-p-menu-navigation", + "expected": 2, + "returned": 5, + "missed": [ + "pr:10934" + ], + "known_positive_recall": 0.5, + "warm_median_ms": 258.0, + "literal_scan_ms": 3905.4, + "output_bytes": 10759, + "candidates_truncated": true + }, + { + "id": "omarchy-7680-lock-screen-password-reveal-toggle", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 306.1, + "literal_scan_ms": 3863.5, + "output_bytes": 10335, + "candidates_truncated": true + }, + { + "id": "omarchy-8824-orphan-prompt-under-omarchy-update-y", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 257.5, + "literal_scan_ms": 3852.1, + "output_bytes": 11153, + "candidates_truncated": true + }, + { + "id": "omarchy-8590-unset-omarchy-path-in-update-dev", + "expected": 2, + "returned": 4, + "missed": [ + "pr:12860" + ], + "known_positive_recall": 0.5, + "warm_median_ms": 425.8, + "literal_scan_ms": 3855.8, + "output_bytes": 10011, + "candidates_truncated": true + }, + { + "id": "omarchy-7783-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 352.9, + "literal_scan_ms": 3895.7, + "output_bytes": 11834, + "candidates_truncated": true + }, + { + "id": "omarchy-11069-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 353.7, + "literal_scan_ms": 3896.5, + "output_bytes": 11760, + "candidates_truncated": true + }, + { + "id": "omarchy-8429-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 354.7, + "literal_scan_ms": 3909.4, + "output_bytes": 11453, + "candidates_truncated": true + }, + { + "id": "omarchy-10413-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 498.8, + "literal_scan_ms": 3898.5, + "output_bytes": 11419, + "candidates_truncated": true + }, + { + "id": "omarchy-13041-same-lines-same-fix", + "expected": 2, + "returned": 4, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 174.6, + "literal_scan_ms": 3929.3, + "output_bytes": 10112, + "candidates_truncated": true + }, + { + "id": "omarchy-13044-same-lines-same-fix", + "expected": 2, + "returned": 4, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 460.8, + "literal_scan_ms": 3897.6, + "output_bytes": 10230, + "candidates_truncated": true + }, + { + "id": "omarchy-8875-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 394.3, + "literal_scan_ms": 3847.1, + "output_bytes": 11542, + "candidates_truncated": true + }, + { + "id": "omarchy-11235-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 352.9, + "literal_scan_ms": 3866.6, + "output_bytes": 11809, + "candidates_truncated": true + }, + { + "id": "omarchy-11344-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 232.4, + "literal_scan_ms": 3929.8, + "output_bytes": 11946, + "candidates_truncated": true + }, + { + "id": "omarchy-11364-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 383.2, + "literal_scan_ms": 3916.3, + "output_bytes": 11751, + "candidates_truncated": true + }, + { + "id": "omarchy-11893-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 316.4, + "literal_scan_ms": 3847.2, + "output_bytes": 11713, + "candidates_truncated": true + }, + { + "id": "omarchy-11918-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 318.2, + "literal_scan_ms": 3896.8, + "output_bytes": 10884, + "candidates_truncated": true + }, + { + "id": "omarchy-11273-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 320.8, + "literal_scan_ms": 3895.4, + "output_bytes": 11824, + "candidates_truncated": true + }, + { + "id": "omarchy-12585-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 329.0, + "literal_scan_ms": 3906.0, + "output_bytes": 11884, + "candidates_truncated": true + }, + { + "id": "omarchy-10749-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 415.8, + "literal_scan_ms": 3906.0, + "output_bytes": 11880, + "candidates_truncated": true + }, + { + "id": "omarchy-12649-same-lines-same-fix", + "expected": 2, + "returned": 4, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 374.4, + "literal_scan_ms": 3856.3, + "output_bytes": 10111, + "candidates_truncated": true + }, + { + "id": "omarchy-11565-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 314.5, + "literal_scan_ms": 3867.7, + "output_bytes": 11891, + "candidates_truncated": true + }, + { + "id": "omarchy-11933-same-lines-same-fix", + "expected": 2, + "returned": 4, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 370.6, + "literal_scan_ms": 3885.0, + "output_bytes": 10192, + "candidates_truncated": true + }, + { + "id": "omarchy-10159-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 318.5, + "literal_scan_ms": 3872.9, + "output_bytes": 11823, + "candidates_truncated": true + }, + { + "id": "omarchy-9847-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 349.3, + "literal_scan_ms": 3875.7, + "output_bytes": 11800, + "candidates_truncated": true + }, + { + "id": "omarchy-8692-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 249.6, + "literal_scan_ms": 3896.7, + "output_bytes": 11269, + "candidates_truncated": true + }, + { + "id": "omarchy-8414-same-lines-same-fix", + "expected": 2, + "returned": 6, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 167.0, + "literal_scan_ms": 3960.6, + "output_bytes": 11994, + "candidates_truncated": true + }, + { + "id": "omarchy-10877-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 255.2, + "literal_scan_ms": 3867.6, + "output_bytes": 11365, + "candidates_truncated": true + }, + { + "id": "omarchy-8406-same-lines-same-fix", + "expected": 2, + "returned": 4, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 390.9, + "literal_scan_ms": 3867.6, + "output_bytes": 10139, + "candidates_truncated": true + }, + { + "id": "omarchy-11728-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 284.9, + "literal_scan_ms": 3885.1, + "output_bytes": 11551, + "candidates_truncated": true + }, + { + "id": "omarchy-8023-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 320.4, + "literal_scan_ms": 3893.5, + "output_bytes": 11313, + "candidates_truncated": true + }, + { + "id": "omarchy-7833-same-lines-same-fix", + "expected": 2, + "returned": 5, + "missed": [], + "known_positive_recall": 1.0, + "warm_median_ms": 390.2, + "literal_scan_ms": 3888.6, + "output_bytes": 10874, + "candidates_truncated": true + } + ] +}