Build a match candidate from an index defined on an unnested record type by g31pranjal · Pull Request #4641 · FoundationDB/fdb-record-layer

g31pranjal · 2026-09-18T14:55:14Z

Makes an index defined on an UnnestedRecordType usable by the Cascades planner. Builds on #4840, which offers such indexes to the planner, and on #4816, whose compensation change lets a match that stands in for a join compensate only its result.

Changes

Expansion: ValueIndexExpansionVisitor builds the candidate over SyntheticRecordType.expand instead of a scan of stored types, and re-expresses the key over the record that expansion assembles, so that e.g. base.unnesting_0.reviewer becomes explode._0.reviewer, the value a query over the same unnesting flows.
ListKeyExpression: the primary key of a synthetic type is now expanded in KeyExpressionExpansionVisitor and ScalarTranslationVisitor (each position is a record of its child's columns), and covering scans read the columns nested inside it, such as __positions.
Fetch: a candidate over a synthetic type fetches with FetchIndexRecords.SYNTHETIC_CONSTITUENTS; mixing stored and synthetic types in one candidate is rejected.
AbstractDataAccessRule: matches are grouped by the set of for-each quantifiers they compensate rather than requiring exactly one, so a match covering the whole join can become a data access.
Placeholder.translateLeafPredicate passes on whether to simplify the translated value.

Tests

unnested-record-type-indexes.yamsql: covering plans and rows over indexes on unnested types, including names that are not protobuf compliant, arrays under nested structs, structs inside array elements, the primary key in the index key, PartiQL and subquery spellings, equality and range predicates, and the choice between an index on the stored table and one on the synthetic type.
SyntheticRecordTypeExpansionTest, IndexEntryTranslatorEquivalenceTest, and TransactionBoundQueryTest, which now expects the unnested index to be used.

Known issues

The grouped PartiQL query over mv_deep_element (GROUP BY "a"."name"."last") is commented out: it cannot be planned since the explode carries its ordinal. It was also not covering before (Aggregates over an index on an unnested record type never use a covering scan #4720).

An index defined on an `UnnestedRecordType` had no match candidate, so the Cascades planner could never use it. Build one from the graph #4626 expands the type into, and teach the relational layer to declare such a type from SQL: a synthetic table whose constituents come from unnesting a repeated field, the metadata to serialize it, and the DDL to define an index over it. The query side is a plain explode of the same repeated field, and it matches the candidate's explode with ordinality through #4619, whose mapping reaches the element because the candidate flows it as a column of a record constructor. `unnested-record-type-indexes.yamsql` covers the whole path end to end, from declaring the type in a schema template to a covering scan of the index. Squashed from the pre-split branch and re-rooted onto #4619, so that #4626 -> #4651 -> #4619 -> #4641 is linear.

g31pranjal · 2026-10-07T13:19:30Z

+      # no grouping
+      # TODO Issue #4720: every column read is in the index, so this should be a covering scan.
+      - query: SELECT COUNT(*) AS "n"
+                 FROM "restaurant", (SELECT "reviewer", "rating" FROM "restaurant"."reviews") AS SQ
+      - explain: "ISCAN(mv_split <,>) | MAP (_.parent AS _0, _.unnesting_0 AS _1) | AGG (count_star(*) AS _0) GROUP BY () | ON EMPTY NULL | MAP (coalesce_long(_._0._0, 0l) AS n)"
+      - result: [{n: 5}]


Six queries covering an inequality on the leading key column, both sides of the same column bounding one range, and an equality prefix of one and two columns followed by a range on the column after it. Each is served by a covering scan of the unnesting index with no residual filter, which is what the bounds being pushed into the scan range looks like.

… function explode Explains now read `TF explode(..)` and navigate `q1._0`, and a query that cannot be planned reports UNABLE_TO_PLAN, as it does on main.

…or checkstyle

…cannot be planned after the explode carries its ordinal

github-actions · 2026-11-10T18:35:09Z

📊 Metrics Diff Analysis Report

Summary

New queries: 47
Dropped queries: 0
Plan changed + metrics changed: 11
Plan unchanged + metrics changed: 4

ℹ️ About this analysis

This automated analysis compares query planner metrics between the base branch and this PR. It categorizes changes into:

New queries: Queries added in this PR
Dropped queries: Queries removed in this PR. These should be reviewed to ensure we are not losing coverage.
Plan changed + metrics changed: The query plan has changed along with planner metrics.
Metrics only changed: Same plan but different metrics

The last category in particular may indicate planner regressions that should be investigated.

New Queries

Count of new queries by file:

yaml-tests/src/test/resources/unnested-record-type-indexes.metrics.yaml: 47

Plan and Metrics Changed

These queries experienced both plan and metrics changes. This generally indicates that there was some planner change
that means the planning for this query may be substantially different. Some amount of query plan metrics change is expected,
but the reviewer should still validate that these changes are not excessive.

Total: 11 queries

Statistical Summary (Plan and Metrics Changed)

task_count:

Average change: -5.0
Average regression: +7.7
Median change: -4
Median regression: +6
Standard deviation: 9.5
Standard deviation of regressions: 2.4
Range: -20 to +11
Range of regressions: +6 to +11
Queries changed: 11
Queries regressed: 3

task_count % change distribution (11 queries, bin = 2%):

     Range                                  n
----------  ------------------------------  ---
[-8%, -6%)  ████                            1
[-6%, -4%)                                  0
[-4%, -2%)                                  0
[-2%, +0%)  ██████████████████████████████  7
[+0%, +2%)  ████                            1
[+2%, +4%)                                  0
[+4%, +6%)  █████████                       2

transform_count:

Average change: -1.8
Average regression: +1.0
Median change: -2
Median regression: +1
Standard deviation: 1.6
Standard deviation of regressions: 0.0
Range: -5 to +1
Range of regressions: +1 to +1
Queries changed: 8
Queries regressed: 1

transform_count % change distribution (8 queries, bin = 2%):

     Range                                  n
----------  ------------------------------  ---
[-8%, -6%)  ███████████████████████         3
[-6%, -4%)                                  0
[-4%, -2%)                                  0
[-2%, +0%)  ██████████████████████████████  4
[+0%, +2%)  ████████                        1

transform_yield_count:

Average change: -1.0
Average regression: +1.0
Median change: -1
Median regression: +1
Standard deviation: 1.0
Standard deviation of regressions: 0.0
Range: -2 to +1
Range of regressions: +1 to +1
Queries changed: 6
Queries regressed: 1

transform_yield_count % change distribution (6 queries, bin = 2%):

       Range                                  n
------------  ------------------------------  ---
[-12%, -10%)  ████████                        1
 [-10%, -8%)                                  0
  [-8%, -6%)                                  0
  [-6%, -4%)                                  0
  [-4%, -2%)                                  0
  [-2%, +0%)  ██████████████████████████████  4
  [+0%, +2%)  ████████                        1

insert_new_count:

Average change: -0.9
Average regression: +1.0
Median change: -1
Median regression: +1
Standard deviation: 1.9
Standard deviation of regressions: 0.0
Range: -5 to +1
Range of regressions: +1 to +1
Queries changed: 8
Queries regressed: 3

insert_new_count % change distribution (8 queries, bin = 2%):

       Range                                  n
------------  ------------------------------  ---
[-20%, -18%)  ████████                        1
[-18%, -16%)                                  0
[-16%, -14%)                                  0
[-14%, -12%)                                  0
[-12%, -10%)                                  0
 [-10%, -8%)                                  0
  [-8%, -6%)                                  0
  [-6%, -4%)                                  0
  [-4%, -2%)                                  0
  [-2%, +0%)  ██████████████████████████████  4
  [+0%, +2%)  ████████                        1
  [+2%, +4%)                                  0
  [+4%, +6%)                                  0
  [+6%, +8%)                                  0
 [+8%, +10%)                                  0
[+10%, +12%)                                  0
[+12%, +14%)  ███████████████                 2

There were no queries with significant regressions detected.

Minor Changes (Plan and Metrics Changed)

In addition, there were 11 queries with minor changes.

Only Metrics Changed

These queries experienced only metrics changes without any plan changes. If these metrics have substantially changed,
then a planner change has been made which affects planner performance but does not correlate with any new outcomes,
which could indicate a regression.

Total: 4 queries

Statistical Summary (Only Metrics Changed)

task_count:

Average change: -173.8
Average regression: +11.0
Median change: +11
Median regression: +11
Standard deviation: 225.8
Standard deviation of regressions: 0.0
Range: -542 to +11
Range of regressions: +11 to +11
Queries changed: 4
Queries regressed: 2

task_count % change distribution (4 queries, bin = 2%):

       Range                                  n
------------  ------------------------------  ---
[-52%, -50%)  ███████████████                 1
[-50%, -48%)                                  0
[-48%, -46%)                                  0
[-46%, -44%)                                  0
[-44%, -42%)                                  0
[-42%, -40%)                                  0
[-40%, -38%)                                  0
[-38%, -36%)                                  0
[-36%, -34%)                                  0
[-34%, -32%)                                  0
[-32%, -30%)                                  0
[-30%, -28%)                                  0
[-28%, -26%)                                  0
[-26%, -24%)                                  0
[-24%, -22%)                                  0
[-22%, -20%)                                  0
[-20%, -18%)  ███████████████                 1
[-18%, -16%)                                  0
[-16%, -14%)                                  0
[-14%, -12%)                                  0
[-12%, -10%)                                  0
 [-10%, -8%)                                  0
  [-8%, -6%)                                  0
  [-6%, -4%)                                  0
  [-4%, -2%)                                  0
  [-2%, +0%)                                  0
  [+0%, +2%)  ██████████████████████████████  2

transform_count:

Average change: -52.3
Average regression: +1.0
Median change: +1
Median regression: +1
Standard deviation: 63.3
Standard deviation of regressions: 0.0
Range: -154 to +1
Range of regressions: +1 to +1
Queries changed: 4
Queries regressed: 2

transform_count % change distribution (4 queries, bin = 2%):

       Range                                  n
------------  ------------------------------  ---
[-56%, -54%)  ███████████████                 1
[-54%, -52%)                                  0
[-52%, -50%)                                  0
[-50%, -48%)                                  0
[-48%, -46%)                                  0
[-46%, -44%)                                  0
[-44%, -42%)                                  0
[-42%, -40%)                                  0
[-40%, -38%)                                  0
[-38%, -36%)                                  0
[-36%, -34%)                                  0
[-34%, -32%)                                  0
[-32%, -30%)                                  0
[-30%, -28%)                                  0
[-28%, -26%)                                  0
[-26%, -24%)                                  0
[-24%, -22%)                                  0
[-22%, -20%)                                  0
[-20%, -18%)                                  0
[-18%, -16%)  ███████████████                 1
[-16%, -14%)                                  0
[-14%, -12%)                                  0
[-12%, -10%)                                  0
 [-10%, -8%)                                  0
  [-8%, -6%)                                  0
  [-6%, -4%)                                  0
  [-4%, -2%)                                  0
  [-2%, +0%)                                  0
  [+0%, +2%)  ██████████████████████████████  2

transform_yield_count:

Average change: -25.5
Median change: -9
Standard deviation: 16.5
Range: -42 to -9
Queries changed: 2
No regressions! 🎉

insert_new_count:

Average change: -28.5
Average regression: +1.0
Median change: +1
Median regression: +1
Standard deviation: 39.4
Standard deviation of regressions: 0.0
Range: -95 to +1
Range of regressions: +1 to +1
Queries changed: 4
Queries regressed: 2

insert_new_count % change distribution (4 queries, bin = 2%):

       Range                                  n
------------  ------------------------------  ---
[-62%, -60%)  ███████████████                 1
[-60%, -58%)                                  0
[-58%, -56%)                                  0
[-56%, -54%)                                  0
[-54%, -52%)                                  0
[-52%, -50%)                                  0
[-50%, -48%)                                  0
[-48%, -46%)                                  0
[-46%, -44%)                                  0
[-44%, -42%)                                  0
[-42%, -40%)                                  0
[-40%, -38%)                                  0
[-38%, -36%)                                  0
[-36%, -34%)                                  0
[-34%, -32%)                                  0
[-32%, -30%)                                  0
[-30%, -28%)                                  0
[-28%, -26%)                                  0
[-26%, -24%)                                  0
[-24%, -22%)  ███████████████                 1
[-22%, -20%)                                  0
[-20%, -18%)                                  0
[-18%, -16%)                                  0
[-16%, -14%)                                  0
[-14%, -12%)                                  0
[-12%, -10%)                                  0
 [-10%, -8%)                                  0
  [-8%, -6%)                                  0
  [-6%, -4%)                                  0
  [-4%, -2%)                                  0
  [-2%, +0%)                                  0
  [+0%, +2%)  ██████████████████████████████  2

insert_reused_count:

Average change: -3.0
Median change: -3
Standard deviation: 0.0
Range: -3 to -3
Queries changed: 1
No regressions! 🎉

There were no queries with significant regressions detected.

Minor Changes (Only Metrics Changed)

In addition, there were 4 queries with minor changes.

g31pranjal added enhancement New feature or request DO NOT MERGE do not merge labels Sep 18, 2026

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch from c781f27 to f3be10b Compare September 18, 2026 15:12

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from 9a0ac5f to 6db6c25 Compare September 18, 2026 15:12

g31pranjal mentioned this pull request Sep 18, 2026

Flow the array element of an explode wrapped in a struct #4623

Closed

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch from f3be10b to 10fe834 Compare September 22, 2026 13:13

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from 6db6c25 to 758a824 Compare September 22, 2026 13:13

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch 6 times, most recently from 647841e to a130c4c Compare September 28, 2026 15:50

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from 758a824 to 9e55929 Compare September 28, 2026 15:50

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch from a130c4c to e35f8bf Compare September 28, 2026 16:02

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch 3 times, most recently from 7a12faa to 8f381f5 Compare October 1, 2026 13:56

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch from e35f8bf to 5e8fdae Compare October 1, 2026 14:55

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from d4f7b19 to 7edbf34 Compare October 1, 2026 14:55

g31pranjal marked this pull request as ready for review October 1, 2026 15:07

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from 7edbf34 to a1ea13f Compare October 2, 2026 14:49

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch from 5e8fdae to 7c96f4f Compare October 2, 2026 14:49

g31pranjal requested review from normen662 and robert-brunel October 6, 2026 07:45

g31pranjal commented Oct 7, 2026

View reviewed changes

g31pranjal mentioned this pull request Oct 7, 2026

Expand an unnested record type into a Cascades graph #4626

Merged

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from 2e57b8a to 60b348d Compare October 8, 2026 13:51

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from eec4cb0 to 4928482 Compare November 5, 2026 00:48

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch from 1dd2445 to d1fa5eb Compare November 5, 2026 09:32

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from 4928482 to 2f6ce9c Compare November 5, 2026 09:32

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch from d1fa5eb to 018c42f Compare November 5, 2026 11:34

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from 2f6ce9c to da3fca9 Compare November 5, 2026 11:34

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch 3 times, most recently from 94e50c8 to 1685a5f Compare November 7, 2026 00:17

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from da3fca9 to b483614 Compare November 7, 2026 01:30

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch from 1685a5f to e5b7282 Compare November 7, 2026 01:31

g31pranjal mentioned this pull request Nov 9, 2026

Replace ExplodeExpression with a table function over an ExplodeValue #4816

Open

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch from e5b7282 to a3f777d Compare November 9, 2026 23:05

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from b483614 to 23fc442 Compare November 9, 2026 23:05

g31pranjal added the DO NOT MERGE do not merge label Nov 10, 2026

g31pranjal mentioned this pull request Nov 10, 2026

Cover indexes on unnested record types with several unnestings and other definitions in yaml-tests #4870

Open

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-ordinality-matching branch from a3f777d to b0a515c Compare November 10, 2026 16:17

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from 23fc442 to 37d978f Compare November 10, 2026 16:17

g31pranjal added 11 commits November 10, 2026 18:30

changes

883f137

changes

71e6b67

more changes

21a68cc

more tests

1f8f04a

nit

8dba6e2

issue

4d26935

more tests

b5b5d6b

Regenerate the unnested record type index expectations over the table…

7594200

… function explode Explains now read `TF explode(..)` and navigate `q1._0`, and a query that cannot be planned reports UNABLE_TO_PLAN, as it does on main.

Indent the ternary continuations of the synthetic candidate helpers f…

86e0145

…or checkstyle

Disable the grouped PartiQL query over the deep-element index, which …

d3d906f

…cannot be planned after the explode carries its ordinal

g31pranjal force-pushed the apple/g31pranjal/unnested/syn-relational-integration branch from 37d978f to d3d906f Compare November 10, 2026 18:33

Navigation Menu

Repository navigation

Build a match candidate from an index defined on an unnested record type - #4641

Build a match candidate from an index defined on an unnested record type#4641
g31pranjal wants to merge 11 commits into
apple/g31pranjal/unnested/explode-ordinality-matchingfrom
apple/g31pranjal/unnested/syn-relational-integration

g31pranjal commented Sep 18, 2026 •

edited

Loading

Uh oh!

g31pranjal Oct 7, 2026

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

g31pranjal commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Known issues

Uh oh!

g31pranjal Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 10, 2026

📊 Metrics Diff Analysis Report

Summary

New Queries

Plan and Metrics Changed

Statistical Summary (Plan and Metrics Changed)

Minor Changes (Plan and Metrics Changed)

Only Metrics Changed

Statistical Summary (Only Metrics Changed)

Minor Changes (Only Metrics Changed)

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

g31pranjal commented Sep 18, 2026 •

edited

Loading