Enable filtered search on mutable HNSW indexes

[xiangfu0]

Summary

MutableVectorIndex could not restrict its search to a document set, so every vector
query on a consuming segment of an upsert table fell back to a brute-force exact scan
over the forward index. This implements FilterAwareVectorIndexReader so those queries
use filtered ANN instead.

• Store the Pinot document ID as a numeric doc value rather than a stored field, so it
  can drive filtered graph traversal and translate hits back from the same reader
  generation used for search.
• Take the document ID from the caller instead of an internal counter. MutableIndex.add
  documents that rows may arrive in any order, so the counter produced wrong document IDs
  under out-of-order ingestion — a latent bug independent of filtering.
• Open the reader from the writer so completed additions are searchable without waiting
  for a commit.
• Give each instance its own temporary directory, so replicas of the same segment hosted
  in one JVM cannot collide on a Lucene write lock. The segment and column stay in the
  directory name so one left behind by a crashed process can still be attributed.

No planner change is required. FilterPlanNode already routes a required candidate
document set to filtered ANN whenever the reader advertises the capability, and to
ExactVectorScanFilterOperator when it does not, so this change alone flips consuming
segments of upsert tables off the exact-scan path.

Rebased

This was a cumulative draft stacked on #19287. That work has since landed as
#19297, #19298, #19299 and #19300, so this branch is now rebased directly onto master
and contains only the mutable-index layer: 2 files, +383/-31, down from 15 files.

Dropped as obsolete during the rebase:

• all pinot-core plumbing (VectorCandidateScope, FilterPlanNode,
  VectorSimilarityFilterOperator, ExactVectorScanFilterOperator,
  VectorSearchStrategy) — superseded by the merged design, which reaches the same
  outcome with no changes here;
• the org.jetbrains:annotations dependency added to pinot-segment-local, which was
  unused (the module compiles without it).

Publication boundary

Opening the reader from the writer makes rows visible to search as soon as they are
added, which is slightly ahead of when the segment publishes them to queries (a row is
added to the indexes, then the document count is raised). Documents past that boundary
cannot produce wrong data: BitmapDocIdIterator stops at the segment's numDocs, so
they are never read. The residual effect is that such a document can occupy a top-K slot
and then be dropped, so a query against a consuming segment can return fewer than K rows
within that window.

The window is narrow and the same shape as the pre-existing one (a commit could already
land between a row being added and the count being raised), but it is wider now.
Bounding candidate traversal by the published watermark would close it; testing
pinotDocId < numPublishedDocs inside the filter iterator is O(1) per candidate and
needs no allocation, unlike materializing a dense bitmap per query.

Follow-ups worth considering

• Reuse the existing HnswVectorIndexReader.RoaringBitmapFilterQuery instead of the
  second filter-query implementation added here; extracting a shared base would keep the
  two from drifting.
• A near-real-time reader is opened per search. Lucene's SearcherManager /
  ReaderManager with maybeRefresh() is the usual pattern and is worth benchmarking
  now that this is the default path for realtime vector queries.

Validation

• MutableVectorIndexTest: 10 tests passed, covering supplied document IDs, filtered
  exclusion of nearer disallowed documents, near-real-time visibility of uncommitted
  additions, same-reader-generation translation, directory isolation, and the bitmap copy
  before async dispatch.
• HnswVectorIndexCreatorTest: 6 tests passed.
• Spotless, Checkstyle, and license checks passed on pinot-segment-local.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 66.21622% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.52%. Comparing base (a4ed3a0) to head (e4ab80c).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...local/realtime/impl/vector/MutableVectorIndex.java	66.21%	19 Missing and 6 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19336      +/-   ##
============================================
- Coverage     67.54%   67.52%   -0.02%     
  Complexity     1430     1430              
============================================
  Files          3486     3486              
  Lines        224043   224105      +62     
  Branches      35353    35365      +12     
============================================
- Hits         151339   151338       -1     
- Misses        60674    60744      +70     
+ Partials      12030    12023       -7     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	67.52% <66.21%> (-0.02%)	⬇️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.52% <66.21%> (-0.02%)	⬇️
unittests	67.52% <66.21%> (-0.02%)	⬇️
unittests1	57.61% <0.00%> (-0.03%)	⬇️
unittests2	39.31% <66.21%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xiangfu0]

Review — mutable HNSW filtered search

Reviewed at e4ab80c90a (2 files, +393/−31), against master now that #19297–#19300 have landed. The restack to just the mutable-index layer is a big improvement over the earlier cumulative version, and the stray org.jetbrains:annotations dependency is gone.

What is good

• Fixes a latent bug. The old code stored _nextDocId++, an index-internal counter, but MutableIndex.add(..., docId) documents that "rows can be added in no particular order, so the docId is required". Storing the supplied doc id is correct; the counter would mis-map doc ids under out-of-order ingestion.
• translateTopDocs sorts hits by Lucene doc id before advanceExact — necessary, since NumericDocValues only advances forward. Easy to get wrong; it is right here.
• Translating through doc values (rather than stored fields) keeps the per-hit cost low.
• isCacheable=false plus identity-based equals is the correct pairing for a query-scoped bitmap — it keeps a per-query filter out of Lucene's query cache.
• Per-instance index directories fix Lucene write-lock collisions between same-JVM replicas.
• Tests cover the genuinely risky paths: NRT visibility, same-generation translation, bitmap copy before async dispatch, directory isolation.

1. (Major) NRT reading has no publication bound

DirectoryReader.open(_indexWriter) makes rows visible as soon as they are added to Lucene — including rows the segment has not yet published to queries. Nothing bounds them: translateTopDocs applies no watermark, and _numDocs is used only for commit cadence and debug stats.

An earlier revision of this PR bounded exactly this, via VectorCandidateScope.forMutableSegment(upsertDocIds, numDocs, ...). That bound lived in the base-fix half, which is now merged in a different form, and nothing replaced it here.

Where it bites: on a non-upsert realtime table FilterPlanNode supplies no required doc ids, so the search is unfiltered and the result set is unbounded. A returned doc id at or beyond the numDocs watermark the plan captured is out of range for everything downstream. Upsert tables are protected only incidentally, because the snapshot happens to bound them.

Suggested fix: no bitmap needed. Capture the watermark and drop hits at or above it in translateTopDocs, or test pinotDocId < numPublishedDocs inside FilteredDocIdSetIterator — an O(1) check per candidate and no allocation. (Bounding in the iterator preserves top-K; bounding after selection can return fewer than K.)

2. (Major) Duplicates a filter query that already exists

HnswVectorIndexReader.RoaringBitmapFilterQuery already implements bitmap-filtered traversal for the immutable reader. This PR adds a second, private implementation for the mutable one. Two copies of Lucene filter-iterator logic will drift.

#19303 already solves this: it extracts BasePinotDocIdBitmapFilterQuery and deletes 72 lines from HnswVectorIndexReader, leaving one implementation both readers extend. Adopting that extraction here would be strictly better than a second private copy.

3. (Major) An NRT reader is opened on every search

DirectoryReader.open(_indexWriter) runs per query. Lucene's guidance is SearcherManager/ReaderManager with maybeRefresh(). This was noted as deferred while the fallback was rare, but this PR makes filtered mutable search the default path for realtime vector queries, so the per-search reader is now on the hot path. Worth a benchmark, or an issue to track, rather than silence.

4. (Minor) Temporary directories lose attribution

Files.createTempDirectory("pinot-mutable-vector-") replaces the old <tmp>/<segment>/<column> layout. Isolation is the right fix, but after a JVM crash the orphaned directories can no longer be traced to a segment or column. Including a sanitized segment and column name in the prefix keeps both properties.

---

🤖 Automated review by Claude Code

[xiangfu0]

Superseded by #19303, closing.

#19303 now covers the same capability — making the mutable HNSW index filter-aware so consuming segments of upsert tables use filtered ANN instead of the exact-scan fallback — rebased onto master after #19297–#19301 landed, so it no longer carries the base-fix half this PR was stacked on.

The two ideas from here that were worth keeping are folded in:

• a single doc-id mechanism (numeric doc values drive both filtered traversal and hit translation, dropping the redundant stored field per document);
• per-instance index directories, so two replicas of one segment in the same JVM cannot collide on the Lucene write lock — kept segment and column in the path so a leaked directory is still attributable.

Two things were deliberately not carried over. This branch reimplements the Lucene filter iterator privately, while HnswVectorIndexReader already has one; #19303 extracts a shared base class instead, so there is one implementation rather than three. And this branch opens a fresh DirectoryReader from the writer on every search; #19303 refreshes a shared SearcherManager and only on the filtered path, leaving the unfiltered path on the cheaper last-committed view.

Also noted for whoever revisits this area: the org.jetbrains:annotations dependency added to pinot-segment-local/pom.xml here is unused in that module.