Add DistributeConjunctsIntoOrFilterOptimizer to push AND conjuncts into OR branches - #19350
Open
waterWang wants to merge 1 commit into
Open
Add DistributeConjunctsIntoOrFilterOptimizer to push AND conjuncts into OR branches#19350waterWang wants to merge 1 commit into
waterWang wants to merge 1 commit into
Conversation
…to OR branches (apache#19339) When a filter has the shape `P AND (A OR B)`, Pinot evaluates the `(A OR B)` subtree independently of `P`. If a branch of the OR contains a predicate with no index (e.g. IN_SUBQUERY evaluated via ExpressionScanDocIdIterator), that predicate is evaluated for every document matching the branch, not only for the documents that satisfy `P`. This optimizer distributes selective conjuncts (EQUALS/IN on single columns) from an enclosing AND into each OR branch: P AND (A OR B) → P AND ((P AND A) OR (P AND B)) This is sound because P ∧ (A ∨ B) ≡ P ∧ ((P ∧ A) ∨ (P ∧ B)) in three-valued logic when the filter only passes TRUE (SQL WHERE semantics). The optimizer runs after the merge optimizers (MergeEqInFilterOptimizer, MergeRangeFilterOptimizer) so that merges apply to the original AND/OR structure before distribution. Fixes apache#19339
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #19339
Description
When a filter has the shape
P AND (A OR B), Pinot evaluates the(A OR B)subtree independently ofP. If a branch of the OR contains a predicate with no index (e.g.IN_SUBQUERYevaluated viaExpressionScanDocIdIterator), that predicate is evaluated for every document matching the branch, not only for the documents that satisfyP.This PR adds a new
DistributeConjunctsIntoOrFilterOptimizerthat distributes selective conjuncts (EQUALS/IN on single columns) from an enclosing AND into each OR branch:This rewrite is sound because in three-valued logic,
P ∧ (A ∨ B) ≡ P ∧ ((P ∧ A) ∨ (P ∧ B))when the filter only passes TRUE (SQL WHERE semantics).The bug in detail
The
AndDocIdSet#iterator()method chooses between two strategies at the eager/lazy threshold. When an OR subtree contains an AND with both index-based and scan-based children, the eager path materializes the index-based children into a bitmap over the entire segment before applying the scan-based predicate. This means expensive predicates likeIN_SUBQUERYare evaluated for every document in the segment, not just those matching the enclosing AND's selective predicateP.Adding an index to a column inside an OR branch makes the query slower because it triggers the eager path:
In production, the same query returning the same two rows went from ~500 to ~17,000,000
numEntriesScannedInFilterafter adding indexes to two columns in the OR branch.Test plan
DistributeConjunctsIntoOrFilterOptimizerTestwith 5 test cases:P AND (A OR B)→P AND ((P AND A) OR (P AND B))