Enable filtered vector search on mutable HNSW and share the doc-id filter query

[xiangfu0]

Summary

The FULL-upsert vector candidate-generation fix (#19297–#19300) makes FilterPlanNode
select filtered ANN whenever a reader advertises the capability, and fall back to a
correctness-first exact scan when it does not. Consuming segments of upsert tables are
left on that fallback, because the mutable index cannot restrict its search. This PR makes
it one of the readers that can.

Mutable index becomes filter-aware. MutableVectorIndex stores the supplied Pinot
document ID as a numeric doc value alongside the existing stored field, translates hits
through it, and reports supportsPreFilter() == true. Consuming segments therefore use
filtered ANN rather than a brute-force scan. No planner change is needed — master
already routes on that capability.

Directory isolation. Each index instance gets a private directory under a
segment-named parent, so two replicas of one segment hosted in the same JVM cannot collide
on the Lucene write lock. Segment and column stay in the path, so a leaked directory can
still be attributed to its owner.

One Lucene filter iterator instead of two. BasePinotDocIdBitmapFilterQuery is
extracted so the mutable index and HnswVectorIndexReader share a single implementation;
HnswVectorIndexReader.RoaringBitmapFilterQuery now extends it (−76 lines there).

supportsPreFilter() becomes a hard contract. A reader returning true commits to
returning a strict subset of the supplied bitmap on every filtered call, and must never
heuristically degrade to unfiltered search — the engine relies on filtered search to keep
obsolete row versions out of candidate generation. Readers that can only honor the filter
conditionally must return false and take the exact-scan fallback. The previous wording
explicitly invited the conditional behavior this now forbids.

Consuming-segment expectations updated. VectorUpsertTableTest (added in #19301)
asserted that consuming segments fall back to the exact scan because the mutable index
was not filter-aware. That is exactly what this changes, so those assertions now require
filtered ANN on the mutable index instead.

Behavior change (release notes)

Consuming segments of tables with an HNSW vector index switch from a brute-force exact scan
to filtered ANN. This is on by default and has no opt-out: vector search on upsert tables is
new, and the exact-scan fallback was a stopgap for the mutable index not honoring filters,
not behavior worth preserving. Top-K on those segments becomes approximate, as it already is
on sealed segments.

FilterAwareVectorIndexReader#supportsPreFilter() now defaults to false and is a hard
contract: returning true commits to always returning a subset of the supplied bitmap, with
no heuristic degradation to unfiltered search. The previous Javadoc explicitly sanctioned
conditional filtering, so an out-of-tree reader written against it must now either opt in
deliberately or keep the safe default and take the exact-scan path. All five in-tree readers
opt in explicitly.

Validation

All commands ran with JDK 25 against current master.

• pinot-segment-local: 26 tests (MutableVectorIndexTest, HnswFilterAwareUpsertTest,
  VectorIndexTypeTest, HnswVectorIndexCreatorTest).
• pinot-core: 572 tests, unchanged by this PR but run as a regression check, including
  TableIndexingTest (506) and the vector filter/planner suites.
• VectorUpsertTableTest: 2 query-engine invocations pass, now exercising the
  filtered-ANN path on the consuming segment.
• Spotless, Checkstyle, and license checks pass on the touched modules.

Relationship to #19336

#19336 implemented the same mutable-index capability on top of the pre-split #19287. This PR
supersedes it, and #19336 is closed. The two ideas worth keeping from it are folded in here:
a single doc-id mechanism (numeric doc values for both filtering and hit translation, dropping
a redundant stored field per document) and per-instance directory isolation. The rest of this
PR's implementation is smaller than #19336's for the same capability, reuses the existing
HnswVectorIndexReader filter query rather than adding a third copy, and refreshes a shared
SearcherManager instead of opening a fresh reader on every query.

Scope notes

This PR touches no pinot-core production code — the required-document plumbing it
originally carried is already in master via #19300, so it was dropped rather than
duplicated. Also dropped while rebasing: VectorSearchSpec (master takes these as
constructor arguments, per review feedback on #19297), a duplicate VectorUpsertTableTest
(#19301 owns it), duplicate ExactVectorScanFilterOperatorTest cases, a new validation
rejecting vector indexes on dictionary-encoded columns (an independent, arguably
backward-incompatible config change that broke the existing dict_mv_FLOAT,vector_index
case — worth its own PR), and a VECTOR_SIMILARITY_RADIUS candidate restriction (a
performance optimization unrelated to this PR's purpose; radius results were already
correct via the saturation fallback plus the outer snapshot AND).

[codecov-commenter]

Codecov Report

❌ Patch coverage is 71.69811% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.58%. Comparing base (bbbed25) to head (a69086e).

Files with missing lines	Patch %	Lines
...local/realtime/impl/vector/MutableVectorIndex.java	80.28%	10 Missing and 4 partials ⚠️
...dex/readers/vector/BaseDocIdBitmapFilterQuery.java	48.00%	12 Missing and 1 partial ⚠️
...nt/index/readers/vector/HnswVectorIndexReader.java	66.66%	2 Missing ⚠️
...t/index/readers/vector/IvfPqVectorIndexReader.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19303      +/-   ##
============================================
+ Coverage     57.71%   67.58%   +9.86%     
- Complexity        7     1430    +1423     
============================================
  Files          2686     3487     +801     
  Lines        163987   224234   +60247     
  Branches      26627    35390    +8763     
============================================
+ Hits          94640   151539   +56899     
+ Misses        61352    60667     -685     
- Partials       7995    12028    +4033     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	67.58% <71.69%> (+9.86%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.58% <71.69%> (+9.86%)	⬆️
unittests	67.57% <71.69%> (+9.86%)	⬆️
unittests1	57.66% <0.94%> (-0.05%)	⬇️
unittests2	39.33% <70.75%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xiangfu0]

Review — enforce the upsert doc-ids snapshot in vector candidate generation

Reviewed at aebbdae5f6 (18 files, +2221/−239) against current master.

Context first: half of this PR has already landed

#19297–#19300 merged the base fix, so ExactVectorScanFilterOperator, VectorSimilarityFilterOperator, FilterPlanNode and their tests are now duplicated by (and will conflict with) master. That part of this PR is done.

What is not in master, and not in #19336 either, is the rest — and it is the most valuable part:

Piece	Status elsewhere
FilterAwareVectorIndexReader contract hardening	nowhere else
BasePinotDocIdBitmapFilterQuery	nowhere else
HnswVectorIndexReader (−72 lines of duplication)	nowhere else
VectorIndexType	nowhere else
VectorRadiusFilterOperator	nowhere else
MutableVectorIndex	overlaps #19336

The recommendation is to reduce this PR to those rows and let it be the mutable-index/SPI PR, rather than close it — closing it as superseded would silently drop the items above.

What is genuinely good

• The SPI hardening is the standout. Turning supportsPreFilter() into an unconditional commitment — with the class-level note that the filter is a correctness contract and an implementation must never heuristically degrade to unfiltered search — closes a real hazard. Under the old "may support it only for certain selectivities" wording, a reader could legally answer true at plan time and degrade at execution time, which is precisely the case where obsolete row versions would silently re-enter top-K. Making it binary, with exact scan as the honest fallback, is the right contract.
• BasePinotDocIdBitmapFilterQuery is the correct factoring. One constant-score weight/scorer, a createLeafIterator hook per reader, isCacheable=false, identity-based equals/hashCode including getClass(). It removes 72 lines from HnswVectorIndexReader instead of adding a second copy — the opposite of what Enable filtered search on mutable HNSW indexes #19336 currently does.
• Stores the supplied Pinot doc id rather than an internal counter, fixing the same latent out-of-order-ingestion bug.
• Keeps DirectoryReader.open(_indexDirectory), so it does not have the unpublished-document exposure that NRT reading introduces in Enable filtered search on mutable HNSW indexes #19336.
• Extending the required-doc-ids restriction to VectorRadiusFilterOperator is unique to this PR, and the Javadoc is commendably honest that radius results were already correct and this is about not burning the candidate budget (and the saturation latency cliff) on obsolete rows.

1. (Blocking, process) Drop the already-merged half

Rebase onto current master and remove the base-fix files. Without that this PR cannot merge, and its diff hides the parts that still need review.

2. (Major) VectorSearchSpec reverses a merged review decision

master deliberately passes the required doc-id bitmap directly. The wrapper type that used to hold it was removed during review of #19297 — the reviewer's point was that a single-field wrapper adds a layer without adding meaning, and a type should be introduced when a second field actually exists. Reintroducing VectorSearchSpec would undo that. If it carries more than one field, that changes the argument — but then the extra state should be the justification, called out explicitly.

3. (Minor) Translate through doc values, not stored fields

translateTopDocs resolves each hit with storedFields.document(scoreDoc.doc), which decompresses a stored-field block per hit, while the numeric doc values needed for filtering are already maintained on the same documents. Reading the doc id from doc values instead is cheaper on the hot path and would let the StoredField be dropped entirely, halving the per-document id storage. (#19336 does the translation this way, and additionally sorts hits by Lucene doc id first, which is required for forward-only advanceExact.)

4. Freshness trade-off worth stating

Committed-only reading means rows added since the last commit are invisible to vector search, so a FULL-upsert query can return fewer than K rows until the next commit fires. That is master's existing behavior and not a regression — but since #19336 takes the opposite approach (NRT, fresher, and needing a publication bound), whichever lands should say which trade-off it is choosing and why.

---

🤖 Automated review by Claude Code

[Jackie-Jiang]

The added test is failing

[Jackie-Jiang]

Name it properly

[xiangfu0]

Renamed to BaseDocIdBitmapFilterQuery. The Pinot prefix was redundant inside the repo, and the neighbouring DocIdTranslator in HnswVectorIndexReader already uses unprefixed DocId for the Pinot side, so this matches what is there. The Lucene-vs-Pinot doc id distinction that motivated the prefix is stated in the class Javadoc instead, which is where it actually needs explaining.

The subclasses keep their existing names, so the family now reads BaseDocIdBitmapFilterQuery -> RoaringBitmapFilterQuery / NumericDocValuesBitmapFilterQuery. Happy to use a different name if you had one in mind.

🤖 Addressed by Claude Code