[travsr-retrieval] conceptual queries miss 7 of 8 — diagnose by stage, then fix (roadmap 04)

[ritikpal1122]

Roadmap step 04 of 6. The long pole, and the only item on the roadmap that becomes a moat rather than a bullet point.

The problem, stated plainly

From CLAUDE.md, our own measurements:

literal queries      6/6 hit          solved
conceptual queries   7/8 MISS         the open problem
salad queries        4/8 leak         precision gap
abstention           3/3 on nonsense  works, but tuned so conservatively
                                      it refuses conceptual queries it should answer

kubernetes/kubernetes: hit@1 0.208, MRR 0.25. Travsr on itself: hit@1 0.292.

Set against the competitive picture: the category leader is an index exposing graph primitives. On structural questions — who calls this, what imports that — they are good, they have 66,000 users, and they install in one command. If Travsr is also just good at structural questions, there is no reason for anyone to switch.

The entire argument for Travsr existing is that it answers questions a graph traversal cannot: where is retry handled, what does auth actually do here. That is conceptual retrieval, and it is the thing that does not work.

This blocks any aggressive positioning. A pitch is downstream of these numbers, not a substitute for them.

What is already known about why

Do not restart from zero; the diagnosis is partly done.

• Abstention is tuned too conservatively and refuses answerable conceptual queries. That is a threshold decision, and it is measurable.
• RFC-025 (RFC-025: execution-path corridor shape and prize function (#527 P1) #654) found the analogous failure in get_execution_path: the selection rule answers the wrong question rather than answering slowly. The corridor is a ball around the source, and no amount of tuning reaches past that. Worth checking whether seed selection has the same shape of defect.
• Semantic embeddings and the cross-encoder reranker are both optional and fail open, so a misconfigured install silently degrades to lexical-only. Confirm which mode the bad numbers were measured in.
• travsr explain and seed_trace exist and are the right instruments. Use them before changing any weights.

Scope

Diagnosis first, then a change with a number attached. Specifically not: tuning constants until the benchmark moves.

• Take the 8 conceptual queries and trace each miss to a stage: seed selection, expansion, ranking, budget, or abstention. Publish the breakdown per query.
• Fix by stage rather than globally. A seed-selection miss and a rank-burial miss have nothing to do with each other and no single knob fixes both.
• Revisit the abstention threshold against the conceptual set specifically, and report false-refusal rate as a separate number from precision.
• Re-run after each change, on both corpora.

Acceptance criteria

• Per-query failure-stage breakdown for all 8 conceptual queries, published
• Conceptual queries clear 6 of 8
• hit@1 moves materially off 0.208 on kubernetes, with the number published either way
• False-refusal rate on the conceptual set reported separately, and reduced
• Salad-query leak rate does not regress while chasing the above
• Every reported figure reproducible by a stated command

Explicitly not acceptable

Reaching the gate by loosening abstention until something is always returned. Precision and false-refusal are reported as separate numbers precisely so that trade cannot be made quietly.

References

• CLAUDE.md, current-phase measurements
• bench/report-*.md
• RFC-025: execution-path corridor shape and prize function (#527 P1) #654 / RFC-025 for the same class of defect in path selection
• travsr explain, seed_trace, embed_knn_probe