fix(rerank): exact-pin flashrank + pin PALACE_RERANK_MODEL — freeze deployed ranking - #204
Conversation
…eployed ranking The determinism probe (#203) proved the /search rerank stage is deterministic for a FIXED model, but flagged a latent risk: requirements.txt pinned `flashrank>=0.2.10` (a floor) and PALACE_RERANK_MODEL was unset, so a fresh deploy could pull a newer flashrank with different bundled ONNX weights / tokenizer — which WOULD silently reorder /search top-5 across that deploy. Freeze the model: - requirements.txt: flashrank>=0.2.10 → flashrank==0.2.10 (the currently deployed version on familiar — no behavior change, just frozen). - palace-daemon.service: add Environment=PALACE_RERANK_MODEL=ms-marco-TinyBERT-L-2-v2 (the code default, now explicit so deployed ranking is byte-stable). - tests/test_rerank.py::TestRerankPinHardening: assert the exact pin + the systemd model env stay in place, so a future floor-pin regression fails CI. No production code touched; deployed behavior unchanged (pins to what's already running). See docs/evals/2026-05-30-retrieval-determinism.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request implements hardening measures to ensure the determinism of the /search reranking stage. By strictly pinning the flashrank dependency and explicitly defining the rerank model in the service configuration, the changes eliminate the risk of silent ranking reorders caused by environment drift during new deployments. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request ensures retrieval determinism across deployments by pinning the flashrank dependency to an exact version (==0.2.10) in requirements.txt and explicitly setting the PALACE_RERANK_MODEL environment variable in palace-daemon.service. It also introduces unit tests to enforce these pins. The feedback suggests specifying encoding="utf-8" when opening these files in the new tests to prevent potential UnicodeDecodeError on systems where the default encoding is not UTF-8.
| """ | ||
|
|
||
| def test_flashrank_is_exact_pinned(self): | ||
| with open(os.path.join(_ROOT, "requirements.txt")) as fh: |
There was a problem hiding this comment.
Both requirements.txt and palace-daemon.service contain non-ASCII characters (such as the em-dash —). Opening these files without specifying an explicit encoding can lead to a UnicodeDecodeError on platforms where the default system encoding is not UTF-8 (e.g., Windows or environments with a restricted locale like LANG=C). Specifying encoding="utf-8" ensures the tests are portable and robust across different environments.
| with open(os.path.join(_ROOT, "requirements.txt")) as fh: | |
| with open(os.path.join(_ROOT, "requirements.txt"), encoding="utf-8") as fh: |
| self.assertNotIn(">=", lines[0]) | ||
|
|
||
| def test_systemd_unit_pins_rerank_model(self): | ||
| with open(os.path.join(_ROOT, "palace-daemon.service")) as fh: |
There was a problem hiding this comment.
Specify encoding="utf-8" when opening palace-daemon.service to prevent potential UnicodeDecodeError on systems where the default encoding is not UTF-8, especially since the file contains non-ASCII characters like the em-dash —.
| with open(os.path.join(_ROOT, "palace-daemon.service")) as fh: | |
| with open(os.path.join(_ROOT, "palace-daemon.service"), encoding="utf-8") as fh: |
Summary
Closes the latent risk flagged by the determinism probe (#203). That probe proved the
/searchrerank stage is deterministic for a fixed model — but the model wasn't fully frozen:requirements.txtpinnedflashrank>=0.2.10(a floor), so a fresh deploy could pull a newer flashrank with different bundled ONNX weights / tokenizer.PALACE_RERANK_MODELwas unset (relying on the code default).Either could silently reorder
/searchtop-5 across a deploy — a real reorder, not a nondeterministic one. This freezes the model.Changes
requirements.txt:flashrank>=0.2.10→flashrank==0.2.10. Pinned to the version currently deployed on familiar (verified read-only) — no behavior change, just frozen. Comment documents the why + "bump deliberately, re-run the determinism test."palace-daemon.service: addEnvironment=PALACE_RERANK_MODEL=ms-marco-TinyBERT-L-2-v2(the existing code default, now explicit) so deployed ranking is byte-stable across deploys.tests/test_rerank.py::TestRerankPinHardening: 2 guard tests asserting the exact pin (==, not>=) and the systemd model env stay in place — a future floor-pin regression fails CI.Safety
No production code touched. Both pins point at what's already running, so deployed behavior is unchanged — this only prevents a future deploy from drifting. flashrank version confirmed
0.2.10on familiar (read-only check); no prod restart, no writes.Tests
tests/test_rerank.py19/19 pass (17 prior incl.TestRerankDeterminism, + 2 new pin guards). Additive.References
docs/evals/2026-05-30-retrieval-determinism.md(the #203 finding).🤖 Generated with Claude Code