Multi-Stage: Add SQL:2016 MATCH_RECOGNIZE (row pattern recognition) - #19311
Open
xiangfu0 wants to merge 3 commits into
Open
Multi-Stage: Add SQL:2016 MATCH_RECOGNIZE (row pattern recognition)#19311xiangfu0 wants to merge 3 commits into
xiangfu0 wants to merge 3 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19311 +/- ##
============================================
+ Coverage 57.71% 67.66% +9.95%
- Complexity 7 1430 +1423
============================================
Files 2686 3499 +813
Lines 163987 225929 +61942
Branches 26627 35762 +9135
============================================
+ Hits 94640 152884 +58244
+ Misses 61352 60838 -514
- Partials 7995 12207 +4212
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
added
feature
xiangfu0
force-pushed
the
claude/match-recognize-oss
branch
from
August 20, 2026 09:03
ededa2b to
3435400
Compare
xiangfu0
commented
Aug 23, 2026
xiangfu0
commented
Aug 23, 2026
xiangfu0
force-pushed
the
claude/match-recognize-oss
branch
3 times, most recently
from
August 26, 2026 09:21
7ebcfdc to
5f27d70
Compare
xiangfu0
commented
Aug 26, 2026
xiangfu0
left a comment
xiangfu0
left a comment
Contributor
Author
There was a problem hiding this comment.
Review of current head 5f27d70c.
The correctness, rolling-upgrade, and resource-behavior issues below should be addressed before merge. I did not duplicate the two existing unresolved threads about v2/lite rejection and _closedPartitionKeys cardinality.
Process follow-ups:
- This is a 56-file, 7.5K-line change spanning protobuf, planning, execution, configuration, and integration. Please either link the reviewed design/maintainer agreement for keeping it as one vertical slice, or split it into reviewable stacked PRs.
- Please link the sender-side-sorting TODO to a tracking issue.
- Both commits contain AI
Co-Authored-Bytrailers; repository guidance says to omit those, so please rewrite the commit messages before merge.
xiangfu0
force-pushed
the
claude/match-recognize-oss
branch
4 times, most recently
from
August 29, 2026 09:07
9623e5c to
ccb8dea
Compare
xiangfu0
added
design-review
Adds row pattern recognition to the multi-stage query engine, following the same shape as the UNNEST support added in apache#17168: a new plan node, an exchange-insertion rule, and an intermediate-stage operator. Pinot's parser already accepted the full MATCH_RECOGNIZE grammar (Parser.jj carries Calcite's Babel production). What was missing was operator-table registration, validation, a plan node, and all of execution. Front end - Register PREV/NEXT/FIRST/LAST/CLASSIFIER/MATCH_NUMBER/RUNNING/FINAL in PinotOperatorTable, which is a strict allow-list. - New MatchRecognizeValidator runs on the SqlNode tree before conversion. It rewrites an OMITTED AFTER MATCH clause to SKIP PAST LAST ROW: Calcite substitutes SKIP TO NEXT ROW, but SQL:2016, Trino, Snowflake and Oracle all default to SKIP PAST LAST ROW. The two differ in whether matches overlap, so a query ported from another engine would otherwise silently return different rows. Conversion erases the omitted-vs-explicit distinction, so the rewrite has to happen here. - Deferred constructs are rejected at planning time with actionable messages rather than silently mis-executing: ALL ROWS PER MATCH, SUBSET, PERMUTE, pattern exclusions, WITHIN, aggregates in DEFINE, NULLS FIRST/LAST, and ORDER BY / PARTITION BY on expressions (the last of which otherwise fail inside SqlToRelConverter with AssertionError or ClassCastException). Plan and wire format - MatchNode at plan.proto tag 19, encoding the pattern as a self-contained recursive RowPattern with a symbol table rather than a RexCall tree, with field numbers reserved for SUBSET and WITHIN. - PatternFieldRef in expressions.proto, so RexPatternFieldRef can no longer degrade to a plain InputRef and produce wrong-but-type-correct results. - PinotMatchExchangeNodeInsertRule hash-distributes on the PARTITION BY keys and prepends them to the sort collation, so rows arrive clustered and the operator can match and flush one partition at a time. A missing PARTITION BY is rejected by default, since it collapses the table onto one worker. Runtime - PatternToNfaCompiler builds an NFA with prioritized transitions (alternation in source order; greedy takes the loop edge first, reluctant the exit edge; {n,m} via counter registers rather than state unrolling), so depth-first traversal yields the SQL:2016 preferred match first. - MatchOperator evaluates DEFINE predicates over a classifier tape supporting PREV/NEXT/FIRST/LAST/CLASSIFIER/MATCH_NUMBER, emits MEASURES, and advances per the AFTER MATCH SKIP mode. - Guardrails throw rather than truncate, since a truncated pattern result is a silently wrong one: maxRowsInMatch, maxStepsPerMatchAttempt, and an empty-cycle guard. v1 covers PARTITION BY, mandatory ORDER BY, MEASURES, ONE ROW PER MATCH, all four AFTER MATCH SKIP modes, the full pattern algebra including reluctant quantifiers and anchors, and single-variable aggregates in MEASURES. MATCH_RECOGNIZE is not yet supported under the v2 physical optimizer or lite mode; queries there are covered by ignore flags rather than silently wrong results.
Upstream now enforces `///` markdown doc comments (JEP 467) over `/** */`
Javadoc via a checkstyle RegexpCheck, and this feature branch predates that
rule. Converts all 168 Javadoc blocks across the 23 MATCH_RECOGNIZE files, and
removes an `org.apache.calcite.sql.SqlLiteral` import that the feature commit
added without ever using (UnusedImports flags it independently).
Formatting only: `{@link X}` becomes `[X]`, `{@link X label}` becomes
`[label][X]`, `{@code x}` becomes a backtick span, `<p>` becomes an empty ///
line, and the remaining HTML becomes its markdown equivalent. No documentation
text was reworded or dropped, and the removed import is the only non-comment
line changed.
Verified: checkstyle reports 0 violations in each of pinot-spi, pinot-common,
pinot-query-planner, pinot-query-runtime and pinot-integration-tests (run
per-module, since a combined reactor stops at the first failure and hides the
rest); `javadoc -Xdoclint:reference,syntax` is clean over the changed sources;
pinot-query-planner 1598 tests and pinot-query-runtime 4611 tests still pass.
Harden planner validation, configuration precedence, wire compatibility, partition ordering, matcher resource bounds, cancellation, and output buffering. Add focused and two-server integration coverage for null semantics, global execution, legacy plans, numeric precision, and retained matcher state.
xiangfu0
force-pushed
the
claude/match-recognize-oss
branch
from
August 29, 2026 10:05
ccb8dea to
b612fab
Compare
Contributor
Author
|
Addressed the process follow-ups on head
Local validation passed: the 342-test focused suite, a 32-test matcher/operator rerun after the final retained-state changes, 13/13 two-server integration cases, the full 63-module |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds SQL:2016 row pattern recognition to Pinot's multi-stage query engine: validation, logical/wire plans, exchange planning, and an intermediate-stage matcher.
Pinot's parser already accepts the
MATCH_RECOGNIZEgrammar through Calcite's Babel parser. This PR adds the missing supported-subset validation and execution path.Design and delivery boundary
The parser validation,
MatchNodewire representation, exchange rule, runtime operator, and expression semantics form one end-to-end query operator and must land atomically. Splitting those layers would either expose syntax that cannot execute or dispatch plans that servers cannot interpret. Existing non-MATCH_RECOGNIZEqueries keep their existing plan and execution behavior; unsupportedMATCH_RECOGNIZEmodes fail closed with actionable validation errors. The PR remains labeleddesign-reviewfor maintainer sign-off before landing.Rollback is likewise atomic: revert the feature commits and stop issuing
MATCH_RECOGNIZEqueries. There is no persisted-data or table-schema migration.Front end and planner
PREV/NEXT/FIRST/LAST/CLASSIFIER/MATCH_NUMBER/RUNNING/FINALin the strictPinotOperatorTableallow-list.MatchRecognizeValidator, including SQL:2016's omittedAFTER MATCHdefault (SKIP PAST LAST ROW) and actionable rejection of deferred constructs such asALL ROWS PER MATCH,SUBSET,PERMUTE, exclusions,WITHIN, aggregates inDEFINE, explicit null ordering, expression partition/order keys, and multi-value partition or aggregate inputs.MatchNodeand a self-contained recursiveRowPattern;PatternFieldRefpreserves pattern-variable identity instead of degrading to a plain input reference.PARTITION BY, then sorts by the exchange's actual partition-key order plus the requested order keys. Queries withoutPARTITION BYrequire the explicitallowMatchRecognizeWithoutPartitionByquery option and execute as one global partition.SET usePhysicalOptimizer = falseselects the supported path.Runtime and resource safety
DEFINEpredicates andMEASURESover a classifier tape, with bounded navigation and exact decimal/integer aggregation behavior.Resource-limit precedence is node hint, query option, server config, then default:
maxRowsInMatchPartitionmax_rows_in_match_partitionpinot.query.match.max.rows.per.partitionmaxStepsPerMatchAttemptmax_steps_per_match_attemptpinot.query.match.max.steps.per.attemptScope
v1 covers
PARTITION BY, mandatoryORDER BY,MEASURES,ONE ROW PER MATCH, all four supportedAFTER MATCH SKIPmodes, pattern alternation/concatenation/quantifiers/anchors, and single-variable aggregates in measures.Rolling-upgrade boundary
MatchNodeis plan oneof field 19. Older servers preserve that protobuf field as unknown but see the node oneof as unset (NODE_NOT_SET), so they cannot execute a dispatchedMATCH_RECOGNIZEstage. Upgrade all servers before issuing these queries; there is no mixed-version fallback. Existing query node kinds remain wire-compatible and are unaffected.Testing
MatchRecognizeIntegrationTest: 13/13 passed against a real two-server cluster, including partition order, cross-segment isolation, global execution withoutPARTITION BY, omitted/explicit skip semantics, and no-null/all-null/mixed-null aggregate behavior with null handling enabled and disabled.test-compilepassed on JDK 25.git diff --checkpassed for all affected modules.