Add null-aware star-tree index

[Jackie-Jiang]

Draft. Two defects found in review are reproduced by failing tests in NullAwareStarTreeQueriesTest. The first needs a fix outside this PR, so this is not ready to merge.

1. Grouping by a dimension that contains nulls throws. NoDictionarySingleColumnGroupKeyGenerator bulk-reads a block's values and only then applies the null bitmap row by row, so the reserved null dictionary id reaches the shared segment dictionary before anything checks whether the row is null, and BaseImmutableDictionary reads past the end of the value buffer. That ordering is general group-by code rather than star-tree code, and is being addressed separately first.
2. An always-true predicate no longer excludes null rows. extractPredicateEvaluatorsMap drops a predicate that is always true over real values, so its column never reaches the null check and the star-tree aggregates rows the predicate should have excluded — a predicate over NULL is UNKNOWN, not true. This predates the PR: master has the same always-true skip and the same key-set-driven null check, and nothing here touches that logic. It still belongs here, since it is a null-handling star-tree defect.

Summary

Adds a second star-tree variant that pre-aggregates with null-handling-on semantics, selected with a new nullHandlingEnabled flag on StarTreeIndexConfig. Today a star-tree folds nulls into the column's default value and counts them, which is right for a null-handling-off query and wrong for a null-handling-on one — so a query that enables null handling can only use a star-tree when nothing it touches is actually null, and otherwise drops to a raw scan.

Both variants can live in the same segment. The flag is part of the builder config's identity, so they survive dedup and one is never reused for the other.

Depends on #19218: excluding nulls from the pre-aggregation only agrees with a raw scan once the aggregation functions themselves skip null rows, which is now true of all of them.

Dimensions

Nulls are stored under a reserved dictionary id — the column's cardinality, one past the last real id — so null rows form their own tree node instead of collapsing into the default value's node. The forward index is widened to cardinality + 1 values and StarTreeLoaderUtils recomputes the bit width to match.

A null vector is written for dimensions as well as metrics: the reserved id is out of range for the segment dictionary the star-tree shares, so the query side has to recognise a null from the vector rather than from the stored id.

Metrics

Null values are excluded from the pre-aggregation. A group left with no non-null input has no aggregated value to store, so the metric keeps a placeholder in the forward index and its nullness lives in the metric's null vector — the same shape a regular column is stored in.

COUNT is the one aggregator that still answers for such a group itself, through ValueAggregator#getAllNullAggregatedValue. It is read back by summing the pre-aggregated column rather than through the null vector, and 0 is exactly the placeholder a null would leave behind, so recording the group in the vector would cost a bit per group and buy nothing. Every other aggregator takes the null default, which keeps an all-null group down to a placeholder plus one null-vector bit rather than a serialized empty sketch, and stops getMaxAggregatedValueByteSize sizing the column for one.

COUNT(column) becomes representable as a stored function-column pair so a null-aware tree can pre-aggregate per-column non-null counts. Config parsing keeps the previous COUNT__col → count__* normalisation for regular trees, so existing table configs are unaffected.

The off-heap builder

The on-heap builder holds Records in memory, where a null metric needs no encoding. The off-heap builder serializes every record to a temporary store and could not express one — it dereferenced the missing value while serializing. Nullness is now held in a per-metric RoaringBitmap keyed by doc id, and the serialized record carries the aggregated type's zero in its place.

Keying on the doc id is safe because records never move: sorting permutes an array of doc ids and compares through it, leaving each record where it was written. The record layout is unchanged, so FixedSizeRecordOffsets' arithmetic still holds.

Query routing

Each query is routed to a tree built in the matching mode. A null-handling-on query may still fall back to a regular tree when none of the columns it touches contains a null value — that is the check that already existed for exactly this case, now extracted so both paths share it.

The resolved function-column pairs travel with the project operator through AggregationInfo, because COUNT resolves differently per tree and the executors must read back the same columns that were projected. Whether a star-tree is in use is read off those pairs being present rather than from a separate flag, so the two cannot disagree.

Compatibility

A deployment that does not opt in is unaffected: an old segment has no null.handling.enabled property and reads false, an old table config defaults the flag to false, and the new mode filter skips nothing when both are false. Same trees built, same tree chosen, same answers.

The behavioural change is confined to segments that opt into a null-aware tree, where a null-handling-on query now gets the same answer a raw scan would give instead of falling back to one.

Tests

NullAwareStarTreeBuilderTest builds a segment whose metric is null for a whole dimension group and covers both that the all-null group survives the build and that the two variants coexist and disagree — a regular tree reports the column default as its minimum and has no null vector, a null-aware one excludes those rows and does.

The build mode is a @DataProvider parameter rather than a random choice. BaseStarTreeV2Test picks between ON_HEAP and OFF_HEAP with RANDOM.nextBoolean() across all its subclasses, which would have made the off-heap gap above a coin-flip flake rather than a failure.

NullAwareStarTreeQueriesTest goes through the query path instead of inspecting what the builder stored, over a segment whose dimension contains nulls — which is where the reserved id has to survive being read back. Both of its tests currently fail, for the two reasons above. Each asserts that fewer documents were scanned than the segment holds, so neither can pass by silently falling back to a raw scan, and a missing group fails rather than reading as 0.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 40.00000% with 198 lines in your changes missing coverage. Please review.
✅ Project coverage is 39.36%. Comparing base (5e914c9) to head (6712bd0).

Files with missing lines	Patch %	Lines
.../org/apache/pinot/core/startree/StarTreeUtils.java	0.00%	57 Missing ⚠️
...pinot/core/operator/query/AggregationOperator.java	0.00%	26 Missing ⚠️
...che/pinot/core/operator/query/GroupByOperator.java	0.00%	23 Missing ⚠️
...aggregation/function/AggregationFunctionUtils.java	4.76%	19 Missing and 1 partial ⚠️
...cal/startree/v2/builder/BaseSingleTreeBuilder.java	73.97%	10 Missing and 9 partials ⚠️
.../index/startree/AggregationFunctionColumnPair.java	0.00%	14 Missing ⚠️
.../startree/v2/builder/OffHeapSingleTreeBuilder.java	62.50%	8 Missing and 4 partials ⚠️
...segment/spi/index/startree/StarTreeV2Metadata.java	0.00%	7 Missing ⚠️
...re/operator/query/FilteredAggregationOperator.java	0.00%	6 Missing ⚠️
...t/core/operator/query/FilteredGroupByOperator.java	0.00%	5 Missing ⚠️
... and 5 more

❗ There is a different number of reports uploaded between BASE (5e914c9) and HEAD (6712bd0). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (5e914c9)	HEAD (6712bd0)
unittests1	1	0
unittests	2	1
java-25	6	5
temurin	6	5

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #19367       +/-   ##
=============================================
- Coverage     67.55%   39.36%   -28.19%     
+ Complexity     1430     1429        -1     
=============================================
  Files          3486     3487        +1     
  Lines        224100   224230      +130     
  Branches      35370    35416       +46     
=============================================
- Hits         151392    88277    -63115     
- Misses        60678   127941    +67263     
+ Partials      12030     8012     -4018     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	39.36% <40.00%> (-28.19%)	⬇️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	39.36% <40.00%> (-28.19%)	⬇️
unittests	39.36% <40.00%> (-28.19%)	⬇️
unittests1	?
unittests2	39.36% <40.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[yashmayya]

@Jackie-Jiang I found some major issues with this implementation.

• Grouping by a null dimension crashes
  • I built a segment with d = {5, null, 7, 7} and m = {10, 20, 30, 40}, added a null-aware star-tree, and ran SELECT d, SUM(m) FROM testTable GROUP BY d with null handling on: Caused by: java.lang.IndexOutOfBoundsException ...NoDictionarySingleColumnGroupKeyGenerator.generateKeysForBlockNullHandlingEnabled
  • The cause is an ordering mismatch. The star-tree stores nulls under the reserved id cardinality, which has no dictionary entry. The group key generator reads the whole block of values first, then applies the null bitmap row by row. So the reserved id reaches the dictionary before anything checks whether the row is null, and the read runs off the end of the buffer. Aggregation-only queries survive because they never turn dict ids into values.
• An always-true filter returns a wrong answer
  • Same segment as above, SUM(m) WHERE d <> 99999 returns an incorrect answer when using the star-tree index.
  • StarTreeUtils.extractPredicateEvaluatorsMap drops the predicate as always true. The column then never reaches the null check, and the star-tree counts the null row.