docs(evals): /search rerank is deterministic across restart — root-cause SME #117 reorder

[jphein]

Summary

SME's deployed-E2E ladder (multipass-structural-memory-eval#117) found that re-querying the same lme_* wings via GET /search before vs after the 2026-05-30 daemon restart reordered 49% of top-5 results on identical-looking data (+32% context). A deterministic reranker reloading the same model shouldn't reorder — this probe nails the mechanism.

Finding: the rerank stage IS deterministic across restart

Code (rerank.py, flashrank/Ranker.py): FlashRank's ONNX pairwise path is a pure function of (query, passage, weights) — no sampling/dropout/RNG at inference; the final ordering is Python's stable Timsort; rerank_hits reconstructs by original index with no dict-iteration or float-tie dependence.

Empirical (scripts/evals/rerank_determinism_probe.py, pinned as tests/test_rerank.py::TestRerankDeterminism):

• In-process, same input reranked 5× → byte-identical order + scores.
• Two fresh processes, each reloading the ONNX model (faithful restart sim) → identical top-5 and scores to <1e-9.

[1] IN-PROCESS REPEAT (5×):     DETERMINISTIC   top-5 = [0, 14, 5, 12, 4]
[2] FRESH-PROCESS RELOAD (×2):  order identical: True   scores identical (<1e-9): True
VERDICT: rerank stage is DETERMINISTIC across restart

Root cause of the #117 reorder: a candidate-set change, not the reranker

The restart picked up #199 + #202, but neither caused the plain-/search reorder: #202 only touches the /search/age-fused handler (not used by plain /search), and #199's kind filter was inert because both runs used kind=all.

The real driver is the data event #199's commit message documents: the 2026-05-29 DB rebackfill re-merged checkpoint drawers into mempalace_drawers (862 confirmed on the live palace). Adding ~862 vectors to the searchable collection changes the ANN neighbours returned for a fixed query at a fixed limit, which feeds the deterministic reranker a different candidate pool → a different reordered top-5. That's the #117 signature exactly.

Ruled out (each checked): rerank nondeterminism (proven deterministic), #202 (wrong code path), the kind filter (kind=all), and any change to the lme_* haystack (filed 2026-05-25, fully populated). → A one-time data migration, not an ongoing regression.

Is deployed retrieval deterministic across restarts? Yes, conditionally

For a fixed collection + fixed reranker model, deployed /search reproduces the same top-5 across restarts. The #117 reorder was the one-time rebackfill.

Latent risk flagged (operator call, not changed here): requirements.txt pins flashrank>=0.2.10 (a floor) and PALACE_RERANK_MODEL is unset. A fresh deploy could pull a newer flashrank with different bundled weights/tokenizer — which would reorder for real. Recommend an exact pin + explicit PALACE_RERANK_MODEL in the systemd Environment=.

Files

• docs/evals/2026-05-30-retrieval-determinism.md — full finding + timeline
• tests/test_rerank.py::TestRerankDeterminism — pins the guarantee (2 tests; skip if flashrank absent)
• scripts/evals/rerank_determinism_probe.py — standalone reproducer

Tests

tests/test_rerank.py 17/17 pass (incl. the 2 new determinism tests); test_rerank* + test_search_rerank_endpoint 26/26 pass. Additive only — no production code touched. All probing was read-only against prod (one /search count query + process metadata); no prod restart, no writes.

🤖 Generated with Claude Code

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request investigates and resolves concerns regarding non-deterministic search results following daemon restarts. By conducting a thorough empirical and code-level analysis, it confirms that the reranking pipeline is deterministic and attributes previous anomalies to upstream data changes. The changes are strictly additive, providing documentation and automated verification tools to prevent and diagnose future regressions.

Highlights

• Determinism Verification: Verified that the rerank stage is deterministic across daemon restarts by proving that the FlashRank ONNX path is a pure function and that Python's Timsort ensures stable ordering.
• Root Cause Analysis: Identified that the observed reordering in SME fix(ops): raise mempalace-db cgroup memory limit to 6 GiB (#102) #117 was caused by a candidate-set change resulting from a one-time database rebackfill, rather than any nondeterminism in the reranker.
• New Testing Infrastructure: Added a standalone determinism probe script and integrated regression tests into the test suite to ensure future rerank stability.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a detailed analysis document, a standalone probe script, and unit tests to verify and guarantee the determinism of the FlashRank rerank stage across daemon restarts. The feedback focuses on improving the robustness of the probe script, specifically by making the subprocess stdout parsing more resilient to diagnostic logs or warnings, and ensuring that the PALACE_RERANK_MAX_LENGTH environment variable is read consistently across both the main and worker process configurations.

[gemini-code-assist]

Parsing the last line of stdout directly using splitlines()[-1] is fragile. If the subprocess prints any warnings, deprecation notices, or diagnostic messages (which ONNX Runtime and ML libraries frequently do to stdout), or if stdout is empty, this will raise an IndexError or JSONDecodeError.

Instead, iterate through the lines in reverse order to find the line containing the JSON array (which starts with [ and ends with ]).

Suggested change

	def _fresh_process_run() -> list[dict]:
	"""Re-exec this script in --worker mode and parse its reranked output."""
	proc = subprocess.run(
	[sys.executable, str(Path(__file__).resolve()), "--worker"],
	capture_output=True, text=True, timeout=300,
	)
	if proc.returncode != 0:
	raise RuntimeError(f"worker failed rc={proc.returncode}: {proc.stderr[-500:]}")
	return json.loads(proc.stdout.strip().splitlines()[-1])
	def _fresh_process_run() -> list[dict]:
	"""Re-exec this script in --worker mode and parse its reranked output."""
	proc = subprocess.run(
	[sys.executable, str(Path(__file__).resolve()), "--worker"],
	capture_output=True, text=True, timeout=300,
	)
	if proc.returncode != 0:
	raise RuntimeError(f"worker failed rc={proc.returncode}: {proc.stderr[-500:]}")
	for line in reversed(proc.stdout.splitlines()):
	cleaned = line.strip()
	if cleaned.startswith("[") and cleaned.endswith("]"):
	return json.loads(cleaned)
	raise RuntimeError(f"No JSON array found in worker stdout. stdout: {proc.stdout[-500:]}")

[gemini-code-assist]

The max_length parameter is hardcoded to 512 here, but in _rerank_once() (line 68) it is dynamically read from the PALACE_RERANK_MAX_LENGTH environment variable. If an operator runs this probe with a custom PALACE_RERANK_MAX_LENGTH set, the in-process repeat check and the fresh-process reload check will run with different configurations, potentially leading to inconsistent results or false failures.

We should read the environment variable consistently in both places.

Suggested change

	ranker = Ranker(
	model_name=os.getenv("PALACE_RERANK_MODEL", "ms-marco-TinyBERT-L-2-v2"),
	max_length=512,
	)
	ranker = Ranker(
	model_name=os.getenv("PALACE_RERANK_MODEL", "ms-marco-TinyBERT-L-2-v2"),
	max_length=int(os.getenv("PALACE_RERANK_MAX_LENGTH", "512")),
	)

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.