[travsr-embed] FUTURE: sidecar GPU/cloud rerank variant (RerankBackend + rerank op) — RFC-021 accelerated follow-up

[Abhishek5517]

Summary

Add a cross-encoder reranker backend to the sidecar: a new rerank(query, [candidate]) -> [score] op that returns an absolute relevance score in [0,1] per (query, candidate) pair. This is the sidecar half of RFC-021 — Cross-Encoder Relevance Arbiter (travsr repo, docs/rfcs/RFC-021-cross-encoder-relevance-arbiter.md); the travsr-mcp seed-selection/confidence integration is tracked separately on the travsr side.

Builds directly on #6. This is a sibling backend, not a new subsystem — it rides the EmbedBackend/BackendFactory capability resolver, the family tag, and the shared encode.rs pipeline that #6 introduces. It should land after (or alongside) #6.

Why

get_context / travsr ask return confidently-labelled irrelevant seeds for broad/off-topic NL queries. Reproduced on the travsr repo with a full arctic-embed-m index:

query: "delete all user accounts and drop the database"
→ confidence: STRONG, 80 nodes
  fn:Guard.drop (Rust Drop impl)     [score 1.00]
  fn:SqliteStore.delete_file          [score 0.82]

Root cause is architectural: the confidence gate approximates relevance with geometric signals (BM25 term overlap + bi-encoder cosine), both token-overlap-driven. A bi-encoder encodes query and candidate separately, so it cannot tell "the user's words are also symbol names" from "the user's intent matches this code" (the RFC-019 comment records BGE scoring RemovePod at cosine 0.751 to GetWarningsForPod purely on shared tokens). A cross-encoder encodes the pair jointly and produces an absolute, model-agnostic relevance score — the one signal the geometric stack lacks.

Model

cross-encoder/ms-marco-MiniLM-L6-v2, tagged family = "bert".

Property	Value	Rationale
Architecture	standard BERT, 6 layers, 22.7M params	Only standard BERT runs on tract. DeBERTa/JinaBERT/ModernBERT rerankers fail at graph load (#6) → unavailable on the default + OCI aarch64 builds. family="bert" → TractFactory.can_run already accepts it; ORT-accelerated for free on CoreML/CUDA builds.
Accuracy	NDCG@10 74.30 (TREC DL19)	Ties L12-v2 (74.31); distilled from L12.
Throughput	~1800 docs/s CPU	2x L12-v2.
Primary artifact	model_fp16.onnx = 45.6 MB	Under the 60 MB budget; best tract compatibility below fp32.
ORT-target artifact	model_int8.onnx = 23 MB	int8 is solid on ORT; per-target artifacts already supported by #6.
Rejected	fp32 (91 MB, over budget); L12 (fp16 ~66 MB → over budget → would force risky int8-on-tract)	—

Precision rule: fp16 on tract targets, int8 on ORT targets. tract's int8 QDQ support is the weakest path; fp16 avoids it while staying under budget.

Architecture (rides #6)

pub trait RerankBackend: Send + Sync {
    /// One batched forward pass over (query, candidate) pairs -> relevance in [0,1].
    fn rerank(&self, query: &str, candidates: &[&str]) -> Result<Vec<f32>>;
    fn backend_name(&self) -> &str;      // "tract", "ort/CoreML", "ort/CUDA"
    fn is_accelerated(&self) -> bool;
}

• Reuses encode.rs tokenize/truncate/pad. The only structural differences from EmbedBackend:
  • input is a pair: [CLS] query [SEP] candidate [SEP] (+ token_type_ids);
  • output head is 1 logit -> sigmoid, not a pooled vector.
• Selected by the [travsr-embed] GPU inference via ORT engine cascade with tract CPU fallback #6 capability resolver keyed on family — no new resolver. A RerankFactory per engine mirrors EmbedFactory (TractRerankFactory allowlist includes bert/minilm; OrtRerankFactory.can_run = true).
• Sequence length capped at 256 tokens. Candidate text = signature + ~10-line skeleton (supplied by the caller). Seq length is the dominant latency lever.
• Batched: all K candidates in one forward pass (batch dim = K), K ~ 20-40.

Sidecar protocol

New op over the existing line protocol (same framing as embed / cancel-sentinel):

REQUEST : {"op":"rerank","query":"<str>","candidates":["<str>", ...]}   // len <= K
RESPONSE: {"scores":[0.83, 0.02, 0.41, ...]}                            // aligned to candidates, [0,1]

One process, one model load — reuses the already-spawned sidecar. No second sidecar.

Distribution / catalog

• New catalog entry (mirrors the embed model catalog) with family = "bert", artifact URLs for model_fp16.onnx (tract targets) and model_int8.onnx (ORT targets), tokenizer vocab.
• Downloaded to ~/.travsr/models/<backend>/ via the existing init flow (travsr side wires a rerank init / bundles default-on per RFC-021 decision).
• model.toml carries family / requires_engine so a model the installed sidecar can't run is refused at selection time ([travsr-embed] GPU inference via ORT engine cascade with tract CPU fallback #6 UX guard pattern).

Acceptance criteria

• RerankBackend trait + rerank op implemented; selected by the [travsr-embed] GPU inference via ORT engine cascade with tract CPU fallback #6 capability resolver via family.
• family="bert" reranker runs on the default tract build and OCI aarch64; ORT-accelerated on CoreML/CUDA builds.
• Default build ships model_fp16.onnx (45.6 MB); ORT builds use model_int8.onnx (23 MB); cross-backend parity >= 0.999 on a fixed pair set.
• rerank(query, candidates) returns scores aligned to input order, in [0,1], batched in one forward pass.
• Shared encode.rs used for tokenization (pair input + token_type_ids); only the head differs from the embed path.
• Startup log states the active rerank backend + EP.
• cargo test green on x86_64 and aarch64.
• A model whose family the installed sidecar can't run is refused at selection time (reuses [travsr-embed] GPU inference via ORT engine cascade with tract CPU fallback #6 capability handshake).

Out of scope

• The travsr-mcp build_seed_set / classify_confidence / ask integration (tracked on the travsr side, RFC-021 sections 6-7).
• Fine-tuning the reranker on code pairs (RFC-021 follow-up).
• MCP-sampling LLM judge (separate, higher-ceiling path).

Ref: RFC-021 (travsr repo). Depends on #6.

[Abhishek5517]

Rescoped 2026-07-17 after Principal Architect end-to-end review (RFC-021 §15).

The ML-in-core boundary was amended (output credibility governs, not runtime location; Invariant #3 intact). As a result, the v1 reranker now runs in-process in the main binary via a new pure-Rust crate travsr-rerank (Reranker trait + TractReranker, CPU tract, deterministic). That path does not need the sidecar, so it unblocks the confident-salad fix from the #6 ORT epic.

This issue is therefore rescoped from "v1 dependency" to a FUTURE GPU/cloud rerank variant behind the same Reranker trait:

• Same model (ms-marco-MiniLM-L6-v2, family="bert") and the same rerank op design, but executed by the ORT/GPU backends from [travsr-embed] GPU inference via ORT engine cascade with tract CPU fallback #6 for the accelerated/cloud (SSE) tier where per-query CPU inference throughput matters.
• The in-process TractReranker remains the default everywhere; this sidecar variant is selected only on accelerated builds / the cloud tier.
• Depends on [travsr-embed] GPU inference via ORT engine cascade with tract CPU fallback #6 (backend capability resolver). Not a blocker for the v1 fix.

v1 in-process work is tracked in travsr#459. Keeping this open as the accelerated follow-up.