Expand an unnested record type into a Cascades graph by g31pranjal · Pull Request #4626 · FoundationDB/fdb-record-layer

g31pranjal · 2026-09-16T11:18:34Z

Adds SyntheticRecordType.expand, which turns a synthetic record type into the graph that assembles its records, and implements it for UnnestedRecordType: a select over the parent record's scan joined with one explode per nested constituent, whose result columns are the constituents plus the __positions field the synthetic primary key needs.

The explodes are created WITH ORDINALITY, because a stored position is exactly the ordinal of the element within its owner's array, and there is no other way to recover it. They ask for 0-based ordinals (the pull request below this one); it then only needs widening from the INT the explode flows to the LONG the Positions message declares.

This is groundwork: nothing calls expand yet. It is what lets a value index defined on a synthetic type become a match candidate, so that a query performing the same unnesting by hand can be answered from the index.

An index defined on an `UnnestedRecordType` had no match candidate, so the Cascades planner could never use it. Build one from the graph #4626 expands the type into, and teach the relational layer to declare such a type from SQL: a synthetic table whose constituents come from unnesting a repeated field, the metadata to serialize it, and the DDL to define an index over it. The query side is a plain explode of the same repeated field, and it matches the candidate's explode with ordinality through #4619, whose mapping reaches the element because the candidate flows it as a column of a record constructor. `unnested-record-type-indexes.yamsql` covers the whole path end to end, from declaring the type in a schema template to a covering scan of the index. Squashed from the pre-split branch and re-rooted onto #4619, so that #4626 -> #4651 -> #4619 -> #4641 is linear.

github-actions · 2026-10-01T14:57:07Z

📊 Metrics Diff Analysis Report

Summary

New queries: 0
Dropped queries: 0
Plan changed + metrics changed: 2
Plan unchanged + metrics changed: 0

ℹ️ About this analysis

This automated analysis compares query planner metrics between the base branch and this PR. It categorizes changes into:

New queries: Queries added in this PR
Dropped queries: Queries removed in this PR. These should be reviewed to ensure we are not losing coverage.
Plan changed + metrics changed: The query plan has changed along with planner metrics.
Metrics only changed: Same plan but different metrics

The last category in particular may indicate planner regressions that should be investigated.

Plan and Metrics Changed

These queries experienced both plan and metrics changes. This generally indicates that there was some planner change
that means the planning for this query may be substantially different. Some amount of query plan metrics change is expected,
but the reviewer should still validate that these changes are not excessive.

Total: 2 queries

Statistical Summary (Plan and Metrics Changed)

task_count:

Average change: -11.0
Median change: -11
Standard deviation: 0.0
Range: -11 to -11
Queries changed: 2
No regressions! 🎉

transform_count:

Average change: -1.0
Median change: -1
Standard deviation: 0.0
Range: -1 to -1
Queries changed: 2
No regressions! 🎉

transform_yield_count:

Average change: -1.0
Median change: -1
Standard deviation: 0.0
Range: -1 to -1
Queries changed: 2
No regressions! 🎉

insert_new_count:

Average change: -1.0
Median change: -1
Standard deviation: 0.0
Range: -1 to -1
Queries changed: 2
No regressions! 🎉

Significant Regressions (Plan and Metrics Changed)

There were 2 outliers detected. Outlier queries have a significant regression in at least one field. Statistically, this represents either an increase of more than two standard deviations above the mean or a large absolute increase (e.g., 100).

yaml-tests/src/test/resources/array-join-at.metrics.yaml:196: EXPLAIN SELECT "id", "val", "at" FROM T1, T1."arr1_nn" AS "val" AT "at" WHERE "at" IN (1, 2)
- old explain: EXPLODE arrayDistinct(@c19) | FLATMAP q0 -> { SCAN([IS T1]) | FLATMAP q1 -> { EXPLODE q1.arr1_nn WITH ORDINALITY | FILTER _._1 + 1 EQUALS q0 AS q2 RETURN (q1.id AS id, q2._0 AS val, q2._1 + 1 AS at) } AS q3 RETURN q3 }
- new explain: SCAN([IS T1]) | FLATMAP q0 -> { EXPLODE q0.arr1_nn WITH ORDINALITY | FILTER _._1 + 1 IN @c19 AS q1 RETURN (q0.id AS id, q1._0 AS val, q1._1 + 1 AS at) }
- task_count: 1056 -> 1045 (-11)
- transform_count: 421 -> 420 (-1)
- transform_yield_count: 52 -> 51 (-1)
- insert_new_count: 115 -> 114 (-1)
yaml-tests/src/test/resources/in-predicate.metrics.yaml:503: EXPLAIN SELECT t.id, n, idx FROM array_table AS t, t.numbers AS n AT idx WHERE idx IN (1, 3)
- old explain: EXPLODE arrayDistinct(@c23) | FLATMAP q0 -> { SCAN([IS ARRAY_TABLE]) | FLATMAP q1 -> { EXPLODE q1.NUMBERS WITH ZERO BASED ORDINALITY | FILTER _._1 + 1 EQUALS q0 AS q2 RETURN (q1.ID AS ID, q2._0 AS N, q2._1 + 1 AS IDX) } AS q3 RETURN q3 }
- new explain: EXPLODE arrayDistinct(@c23) | FLATMAP q0 -> { SCAN([IS ARRAY_TABLE]) | FLATMAP q1 -> { EXPLODE q1.NUMBERS WITH ORDINALITY | FILTER _._1 + 1 EQUALS q0 AS q2 RETURN (q1.ID AS ID, q2._0 AS N, q2._1 + 1 AS IDX) } AS q3 RETURN q3 }
- task_count: 1060 -> 1049 (-11)
- transform_count: 423 -> 422 (-1)
- transform_yield_count: 54 -> 53 (-1)
- insert_new_count: 115 -> 114 (-1)

normen662 · 2026-10-07T11:29:23Z

+    }
+
+    @Nonnull
+    private static List<String> arrayFieldPath(@Nonnull final KeyExpression nestingExpression) {


Can you check if you like this better in MessageHelpers?

Doesn't go with MessageHelpers since it doesn't have to do anything with Message. What about in KeyExpression or NullableArrayTypeUtils? I am not fully convinced though

KeyExpression is probably better if this were to be reused.

or leave it here. your call

normen662 · 2026-10-07T12:14:49Z

+        final NestedConstituent parentConstituent = getParentConstituent();
+        final RecordType parentRecordType = parentConstituent.getRecordType();
+        final Quantifier.ForEach parentQuantifier =
+                Quantifier.forEach(ExpansionVisitor.createBaseRef(metaData.getRecordTypes().keySet(),


wouldn't you want to call a child's record types expand here?

added public GraphExpansion expand(@Nonnull final AccessHint accessHint) to RecordType and call it here.

Adds `SyntheticRecordType.expand`, which turns a synthetic record type into the graph that assembles its records, and implements it for `UnnestedRecordType`: a select over the parent record's scan joined with one explode per nested constituent, whose result columns are the constituents plus the `__positions` field the synthetic primary key needs. The explodes are created `WITH ORDINALITY`, because a stored position is exactly the ordinal of the element within its owner's array, and there is no other way to recover it. They are asked for 0-based ordinals, so that an ordinal *is* the position, and for the `(element, ordinal)` struct to be flowed as a record constructor, so that both are sub-values a query's values can be matched against. The ordinals only need widening from the `INT` the explode flows to the `LONG` the `Positions` message declares -- the types have to agree, or the record the expansion flows is not of the type's own planner type. `RecordMetaData` gains `getPlannerTypeForRecordType(s)` overloads that take already-resolved `RecordType`s. The existing name-based overloads resolve against stored record types only, so they cannot describe a synthetic type, whereas `recordTypesForIndex` hands back the type itself for an index defined on one. The tests cover the graph `UnnestedRecordType.expand` builds -- the parent scan, one explode per nested constituent, the `__positions` record, chained constituents and nullable-array wrappers -- and the record-type-taking planner type overloads, including that they describe a synthetic type, which the name-taking ones cannot. `test_records_nested_chain.proto` is added for the chained case: the existing two-level proto, `test_records_double_nested.proto`, is deliberately recursive, and the planner cannot build a type for a recursive descriptor, so a chain of constituents needs a non-recursive record type to unnest. This is groundwork: nothing calls `expand` yet. It is what lets a value index defined on a synthetic type become a match candidate, so that a query performing the same unnesting by hand can be answered from the index.

normen662 · 2026-10-08T14:20:29Z


+    @Nonnull
+    @API(API.Status.INTERNAL)
+    public GraphExpansion expand(@Nonnull final AccessHint accessHint) {


normen662 · 2026-10-08T14:23:02Z

+    public GraphExpansion expand(@Nonnull final AccessHint accessHint) {
+        final GraphExpansion parentExpansion = getParentConstituent().getRecordType().expand(accessHint);
+        final Map<String, Value> elementValuesByConstituent = new HashMap<>();
+        final GraphExpansion.Builder builder = parentExpansion.toBuilder().removeAllResultColumns();


Why do you remove all columns here? you should leave them and add the ones from here to it.

Sorry, this is something that confused me as well :D - This is taking the child's expansion and using its builder (without resultValue) to add explode quantifiers and the "synthetic" resultValue. Tbh, it's as good as adding quantifiers of the child expansion (as of now). Changed to that!

Implements `SyntheticRecordType.expand` for `JoinedRecordType`, which until now threw. #4626 added `expand` and implemented it for `UnnestedRecordType`; this does the same for joins. A joined record type expands to one scan per constituent and one equality predicate per join: - Each constituent is the expansion of its stored record type, a type-filtered scan, flowed whole as a result column named after the constituent. A type joined to itself gets one scan per copy, each with its own quantifier, so the join can relate the copies. - Each join becomes `left = right`, with each side's key expression translated into a value over its own constituent's quantifier. Nested join fields become nested field values. - A constituent with no join leaves the pair unconstrained, which is the same cross product the joined record type itself produces. Outer-joined constituents are rejected with `UnsupportedOperationException`. An outer join keeps a record that has no partner, and an inner equality between two scans would drop it. This is groundwork: nothing calls `expand` yet. It is what lets an index on a joined record type become a match candidate, so that a query performing the same join by hand can be answered from the index. `UnnestedRecordTypeExpansionTest` is renamed to `SyntheticRecordTypeExpansionTest`, which now tests the expansion of both kinds of synthetic type. Its test expecting a joined type to throw is replaced by tests of a two-way join, a self-join, a three-way cyclic join (one equality per join), a join whose constituents are named in the opposite order to their declaration, a nested join field, a constituent with no join, and outer-join rejection.

g31pranjal added the enhancement New feature or request label Sep 16, 2026

g31pranjal added this pull request to stack #4627 September 16, 2026 11:19

g31pranjal removed this pull request from stack #4627 September 16, 2026 13:16

g31pranjal force-pushed the apple/g31pranjal/unnested/opt-in-zero-based-ordinality branch from 4d2d403 to 9b5aca6 Compare September 16, 2026 13:16

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch from 51f14cb to 40cfe34 Compare September 16, 2026 13:16

g31pranjal added this pull request to stack #4628 September 16, 2026 13:17

g31pranjal removed this pull request from stack #4628 September 16, 2026 13:35

g31pranjal force-pushed the apple/g31pranjal/unnested/opt-in-zero-based-ordinality branch from 9b5aca6 to f1509aa Compare September 16, 2026 13:35

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch from 40cfe34 to 28a0ea8 Compare September 16, 2026 13:35

g31pranjal added this pull request to stack #4629 September 16, 2026 13:36

Base automatically changed from apple/g31pranjal/unnested/opt-in-zero-based-ordinality to main September 16, 2026 18:55

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch 4 times, most recently from 03556e0 to 91b08d7 Compare September 17, 2026 14:22

g31pranjal marked this pull request as ready for review September 17, 2026 14:22

g31pranjal requested a review from normen662 September 17, 2026 14:23

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch from 91b08d7 to b8c1fb4 Compare September 17, 2026 22:34

g31pranjal removed this pull request from stack #4629 September 17, 2026 22:35

g31pranjal added this pull request to stack #4632 September 17, 2026 22:35

g31pranjal removed this pull request from stack #4632 September 17, 2026 22:39

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch from b8c1fb4 to 24551d6 Compare September 17, 2026 22:39

g31pranjal changed the base branch from main to apple/g31pranjal/unnested/zero-based-ordinality September 17, 2026 22:43

g31pranjal added this pull request to stack #4634 September 17, 2026 22:44

g31pranjal removed this pull request from stack #4634 September 17, 2026 22:48

g31pranjal changed the base branch from apple/g31pranjal/unnested/zero-based-ordinality to main September 18, 2026 15:06

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch 2 times, most recently from a74faf7 to da06b69 Compare September 23, 2026 17:27

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch from da06b69 to 3fa86c3 Compare September 28, 2026 15:49

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch from 3fa86c3 to 109de81 Compare September 28, 2026 16:02

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch from 109de81 to d6852b0 Compare October 1, 2026 14:55

g31pranjal changed the base branch from main to apple/g31pranjal/unnested/explode-ordinality-struct-value October 1, 2026 14:58

Base automatically changed from apple/g31pranjal/unnested/explode-ordinality-struct-value to main October 2, 2026 14:41

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch from d6852b0 to 30bc80a Compare October 2, 2026 14:49

g31pranjal requested a review from robert-brunel October 6, 2026 12:19

normen662 requested changes Oct 7, 2026

View reviewed changes

g31pranjal added 4 commits October 8, 2026 12:18

address comments

a12245d

nit

214b59c

Drop the zero-based ordinality argument removed by #4704

0a23c13

g31pranjal force-pushed the apple/g31pranjal/unnested/synthetic-type-expand branch from 25492b0 to 0a23c13 Compare October 8, 2026 13:24

g31pranjal requested a review from normen662 October 8, 2026 13:25

normen662 reviewed Oct 8, 2026

View reviewed changes

address comments

32968ff

g31pranjal requested a review from normen662 October 9, 2026 11:22

normen662 approved these changes Oct 9, 2026

View reviewed changes

Merge branch 'main' into apple/g31pranjal/unnested/synthetic-type-expand

ee97c89

g31pranjal merged commit 3d2e6b3 into main Oct 9, 2026
14 checks passed

g31pranjal deleted the apple/g31pranjal/unnested/synthetic-type-expand branch October 9, 2026 12:20

g31pranjal mentioned this pull request Oct 13, 2026

Expand a joined record type into a Cascades graph #4741

Merged

Navigation Menu

Repository navigation

Expand an unnested record type into a Cascades graph - #4626

Expand an unnested record type into a Cascades graph#4626
g31pranjal merged 6 commits into
mainfrom
apple/g31pranjal/unnested/synthetic-type-expand

g31pranjal commented Sep 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 1, 2026

Uh oh!

Uh oh!

Uh oh!

normen662 Oct 7, 2026

Uh oh!

g31pranjal Oct 7, 2026

Uh oh!

normen662 Oct 8, 2026

Uh oh!

normen662 Oct 8, 2026

Uh oh!

Uh oh!

normen662 Oct 7, 2026

Uh oh!

g31pranjal Oct 7, 2026

Uh oh!

normen662 Oct 8, 2026

Uh oh!

normen662 Oct 8, 2026

Uh oh!

g31pranjal Oct 8, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

g31pranjal commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 1, 2026

📊 Metrics Diff Analysis Report

Summary

Plan and Metrics Changed

Statistical Summary (Plan and Metrics Changed)

Significant Regressions (Plan and Metrics Changed)

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

g31pranjal commented Sep 16, 2026 •

edited

Loading