Let an explode declare the shape of the value it flows by g31pranjal · Pull Request #4650 · FoundationDB/fdb-record-layer

g31pranjal · 2026-09-22T21:13:00Z

An EXPLODE WITH ORDINALS's value is a single opaque QueriedValue, even though it flows 2 pieces of information - element and its position. Nothing inside it is reachable, which is fine for evaluation but it leaves no way to relate the element to anything.

Why `RecordConstructorValue` is needed:

An index utilizing an UnnestedRecordType requires that the SyntheticRecordType is expanded into a candidate. This expansion in turn requires unnesting with ordinality, because the proto descriptor of the UnnestedRecordType, as well as its primary key, carries the position of the element. A query that unnests the same collection but asks for no ordinals must be matched against this candidate, by relating the query's element to the candidate's element: MaxMatchMap has to find the query's value among the candidate's reachable sub-values.

MaxMatchMap descends only into record constructors. An opaque QueriedValue of the struct type cannot be descended into. Having RecordConstructorValue, makes the element a reachable sub-value and the match findable structurally.

This matching happens in #4619; the shape it needs is turned on in #4651.

Why the shape is recorded rather than inferred

The value a plan flows is part of the plan's identity: RecordQueryExplodePlan hashes it, so changing the shape changes the plan hash of every WITH ORDINALITY plan. A plan serialized by an earlier version flows the opaque value and was hashed as such, and it has to keep doing both when a newer version reads it back. Deciding the shape from withOrdinality alone would hence not be backward-compatible.

So the shape becomes state the plan records and serializes. This PR adds that state and leaves it at the value every existing plan flows; #4651 changes what newly planned explodes ask for.

What this PR does

ExplodeExpression and RecordQueryExplodePlan gain a flowsRecordConstructorValue flag.

What the flag decides differs between the two variants. With ordinality the struct is the result type either way, so only the shape of the value changes: an older version that cannot see the flag still executes such a plan identically. Plainly, the struct is the flag's doing — the result type is the element type without it and a one-column struct with it — so the plan produces that struct rather than the bare element. That is why each variant needs its own PR to turn the shape on: #4651 for the ordinality one, and #4623 for the plain one, which reaches into everything that reads what an explode flows.

Nothing asks for new shape yet. That will be enabled in later PRs

Compatibility

The flag is serialized in porto. Absent means the opaque value, which is precisely what a plan serialized before the field existed flows, so such a plan keeps both its shape and its hash.

#4651 made a `WITH ORDINALITY` explode flow its `(element, ordinal)` struct as a record constructor, but a plain explode still flowed the bare element. That left the two shapes incomparable: a plain explode on the query side could not be related to a `WITH ORDINALITY` explode on the candidate side, because `MaxMatchMap` descends into a record constructor but not into an opaque value, so there was nothing below the element to match. Every explode now flows a record -- `(element)` or `(element, ordinal)` -- and the element is reachable as `_0` on either side. The flag #4650 introduced is simply set for both variants. Everything that reads what an explode flows follows from that. `FieldValue.ofFieldNamesAndFuseIfPossible` fuses a field access onto an existing field value instead of nesting one inside the other, and the key expression expansion uses it. The pull-up rules require the child of the value being pulled up to be the quantified object value itself, so a nested access could not be pulled through the explode's quantifier at all. An in-join and an in-union bind an element of the collection under `__corr_«alias»`, not the struct the explode flows, so the plans they put underneath have to be translated: `ExplodeExpression.elementBindingTranslationMap` makes each such alias stand for the element alone, which composes `«alias»._0` away to a plain reference to the binding. Neither rule can take an explode with ordinality as a source, since there is no place to put the ordinal. On the SQL side, a quantifier over an unnesting stands for the struct, so the element is reached through `_0`: `LogicalOperator` unwraps it when it builds the unnesting's attributes, `QuantifierValues` sees through it when resolving an index definition down to the base record, and a star expansion over an unnesting expands the element's fields rather than the struct's. The `AT` ordinal is a property of the unnesting rather than a column of the row, so it is ephemeral: nameable, but not part of a star. `ExplodePlanTest` follows the same way: a plain explode flows a record, so the skip-and-limit tests read the element out of it, and the plans that have to flow the element itself ask for that shape explicitly. The hash of a plain explode moves accordingly; the hash a plan that flows the element itself has always had is pinned alongside it.

An explode `WITH ORDINALITY` flows an anonymous `(element, ordinal)` struct, and the value standing for that struct is a single opaque `QueriedValue`. Nothing inside it is reachable, which is fine for evaluation -- the plan builds the struct from the protobuf descriptor its declared type names, not from the value -- but it leaves no way to relate the element to anything. `MaxMatchMap` descends into record constructors and into nothing else, so an index over an `UnnestedRecordType`, whose candidate unnests with ordinality, cannot be related to a query that unnests the same collection plainly. `ExplodeExpression` and `RecordQueryExplodePlan` now carry a `flowsRecordConstructorValue` flag. Set, the value is a `RecordConstructorValue` whose columns -- the element, and the ordinal for the `WITH ORDINALITY` variant -- are each reachable on their own; unset, it is the opaque value of the whole explode result type. What the flag decides differs between the two variants. With ordinality the struct is the result type either way, so only the shape of the value changes, and the plan produces the same rows whichever shape it flows. Plainly, the struct is the flag's doing: the result type is the element type without it and a one-column struct with it, and the plan produces that struct rather than the bare element. The shape has to be state the plan records rather than something inferred from `withOrdinality`, because the plan hashes the value it flows: a plan serialized by an earlier version flows the opaque value and was hashed as such, and must keep doing both when a newer version reads it back. So the flag is serialized as `PRecordQueryExplodePlan.flows_rcv`, written only when set, and absent means the opaque value -- which is precisely what a plan from before the field existed flows. Nothing sets the flag yet: every constructor that does not name a shape picks the opaque value, so no explode flows anything different and plan hashes and expected plans are untouched. `ImplementExplodeRule` takes the shape off the expression it implements rather than choosing one, so the plan inherits whatever the expression settled on.

robert-brunel · 2026-09-24T10:03:02Z

+        this.flowsRecordConstructorValue = flowsRecordConstructorValue;
+        this.explodeResultType = explodeResultType(elementType, withOrdinality, flowsRecordConstructorValue);
+        this.resultValue = explodeResultValue(elementType, withOrdinality, flowsRecordConstructorValue);
+        Verify.verify(explodeResultType.equals(resultValue.getResultType()));


We’re computing explodeResultType() twice, once here and once in explodeResultValue().

But since this equality holds, we probably don’t have to actually compute and store this.explodeResultType here anymore in addition to this.resultValue.

Also in the RecordQueryExplodePlan class.

I think there is definitely merit to do this, and we should do this but probably at one of the cleanup stages, where type directly yields homogenous value. I'll create a follow cleanup PR that accommodates this.

robert-brunel · 2026-09-24T10:04:43Z

+
    public ExplodeExpression(@Nonnull final Value collectionValue, final boolean withOrdinality,
-                             final boolean zeroBasedOrdinality) {
+                             final boolean zeroBasedOrdinality, final boolean flowsRecordConstructorValue) {


Also verify that zeroBasedOrdinality == true implies flowRecordConstructorValue == true?

This, does not hold currently. While the zeroBasedOrdinality is enforced everywhere by #4633, flowRecordConstructorValue will be enforced by #4651. After that, the implication will hold true - and will enforce that we are really covered everywhere.

constructor, but a plain explode still flowed the bare element. That left the two shapes incomparable: a plain explode on the query side could not be related to a `WITH ORDINALITY` explode on the candidate side, because `MaxMatchMap` descends into a record constructor but not into an opaque value, so there was nothing below the element to match. Every explode now flows a record -- `(element)` or `(element, ordinal)` -- and the element is reachable as `_0` on either side. The flag #4650 introduced is simply set for both variants. Everything that reads what an explode flows follows from that. `FieldValue.ofFieldNamesAndFuseIfPossible` fuses a field access onto an existing field value instead of nesting one inside the other, and the key expression expansion uses it. The pull-up rules require the child of the value being pulled up to be the quantified object value itself, so a nested access could not be pulled through the explode's quantifier at all. An in-join and an in-union bind an element of the collection under `__corr_«alias»`, not the struct the explode flows, so the plans they put underneath have to be translated: `ExplodeExpression.elementBindingTranslationMap` makes each such alias stand for the element alone, which composes `«alias»._0` away to a plain reference to the binding. Neither rule can take an explode with ordinality as a source, since there is no place to put the ordinal. On the SQL side, a quantifier over an unnesting stands for the struct, so the element is reached through `_0`: `LogicalOperator` unwraps it when it builds the unnesting's attributes, `QuantifierValues` sees through it when resolving an index definition down to the base record, and a star expansion over an unnesting expands the element's fields rather than the struct's. The `AT` ordinal is a property of the unnesting rather than a column of the row, so it is ephemeral: nameable, but not part of a star. `ExplodePlanTest` follows the same way: a plain explode flows a record, so the skip-and-limit tests read the element out of it, and the plans that have to flow the element itself ask for that shape explicitly. The hash of a plain explode moves accordingly; the hash a plan that flows the element itself has always had is pinned alongside it.

alecgrieser

Two things: there's maybe one or two test asserts, but the biggest thing before merging is probably deciding whether this needs to be in the hash code. Most of the other comments are more questions. Overall, I think this looks good

alecgrieser · 2026-09-25T09:39:11Z

-        } else {
+    public static Type explodeResultType(@Nonnull final Type elementType, final boolean withOrdinality,
+                                         final boolean flowsRecordConstructorValue) {
+        if (!withOrdinality && !flowsRecordConstructorValue) {


I think the design makes sense for now, but I did want to ask one thing about the future. In the future, for ExplodeExpressions without ordinality, are we going to return an RCV with a single element, or is this going to return just the element? Do we gain something by always wrapping? There may actually be advantages--for example, it may be easier for the planner to reason about a query for just the value that skips the ordinal if the expression is a projection of an expression on both fields

There are a number of places where this PR needed to update the wrapping logic to accommodate the new state of !withOrdinality && flowsRecordConstructorValue. If we were planning on banning that state long term (i.e., only wrapping if there's an ordinal), then in theory we could save some churn.

We will ultimately wrap for both the cases (i.e., with and without ordinals) - this is what I am aiming for. In short , the reason for it is to have homogeneity between the two case.

Let me explain it with what triggered this. We need the ability to match the query-side unnestings with a candidate side graph for leveraging the UnnestedRecordType. Note that the query-side could have EXPLODE with or without ordinals, but the candidate-side will always have it WITH ORIDINAL (because the ordinals form the primary key and descriptor of synthetic type). With this assumption, I figured out that its relatively difficult match a without-ordinal EXPLODE's QueriedValue from query-side to the RCV-wrapped WITH ORDINAL EXPLODE in the candidate-side. The easiest way, I feel, around this is to homogenize the output of the EXPLODE throughout the system so that both sides are actually FieldValue in the RCV. Having said that, I am still in the process of experimentally verify that the matching is correct - that is, we may have to do more changes around this.

Converting bare Value to RCV of that value has lots of ripple effects throughout, hence I am trying to fix those in #4623. Some of these are simply about making the consumer of the EXPLODE to understand the new shape. While others are more extensive, namely ImplementInJoin and ImplementInUnion that worked under the assumption that the "output" of EXPLODE is always the underlying array value. Still making my way through it, but I think doable.

Unfortunately, we will have to take the longer route, if we decide to move away from the plan this PR proposes. I'd love to discuss this more, if you have ideas related to it.

I am also planning to enable this in 2 steps - 1) only for WITH ORDINAL (smaller blast-radius, sufficient to match single unnested UnnestedRecordType-based indexes), and then, 2) all EXPLODEs, (this affects all plans that have EXPLODE).

#4650 gave an explode the flag that decides whether it flows the element and the ordinal as a `RecordConstructorValue` or as one opaque value of the struct type, but left every explode flowing the opaque value. Ask for the record constructor where the SQL layer builds an `AT` unnesting, which is the only place that produces an explode with ordinality. The record-layer constructors keep defaulting to the opaque value, so nothing else changes shape. Both shapes stand for the same data and evaluate identically -- the plan builds its struct from the protobuf descriptor its declared type names -- but the element and the ordinal are now reachable sub-values, which is what lets anything be related to them. The expected plans of the five `AT` queries that also carry an `IN` move, since an `IN` over a value read out of a record is costed differently than one over an opaque value. They move in both directions: `array-join-at`'s `WHERE "at" IN (1, 2)` drops its join over the `IN` list for a single scan with a residual filter, two queries with an `IN` on the primary key keep their operators but move the unnesting across the in-join boundary, and `in-predicate`'s two queries with an `IN` on the unnested value start scanning the table once per `IN` element, which is #4621. No index match is lost, and all results still verify. A plan serialized before the flag existed still deserializes to the opaque value, so it keeps the shape, and the hash, it was planned with. A release that predates #4650 drops the flag instead, so this cannot be released before #4650 is.

> Supersedes #4651, which GitHub marked as merged when this branch was messed up in the stack reordering #4650, now merged, added the `flowsRecordConstructorValue` flag but left every explode flowing the opaque value. This PR turns it on only for `EXPLODE WITH ORDINALS` (`LogicalOperator.generateCorrelatedFieldAccess`). Wrapping the element of a plain explode in a struct of its own is #4623. ### What changes in plans Nothing in `fdb-record-layer-core` changes, and no core test needed updating. What moves is the expected plan of the five `AT` queries that also carry an `IN`: an `IN` over a value read out of a record is costed differently than one over an opaque value. The movement goes in both directions and is worth a look: - `array-join-at:317`, `WHERE "at" IN (1, 2)`: was an explode of the `IN` list joined with a scan, and is now a single scan with a residual `FILTER _._1 + 1 IN @c19`. Cheaper. - `array-join-at:305` and `in-predicate:402`, both `IN` on the primary key: same operators, but the unnesting moves across the in-join boundary — into the in-join in the first, out of it in the second. - `in-predicate:382` and `in-predicate:410`, both `IN` on the unnested value: the `IN` list explode becomes the outer loop with the table scan inside it, i.e. one full scan per `IN` element. This is #4621 — the cost model scores a data access nested under an in-join the same as an unnested one.

constructor, but a plain explode still flowed the bare element. That left the two shapes incomparable: a plain explode on the query side could not be related to a `WITH ORDINALITY` explode on the candidate side, because `MaxMatchMap` descends into a record constructor but not into an opaque value, so there was nothing below the element to match. Every explode now flows a record -- `(element)` or `(element, ordinal)` -- and the element is reachable as `_0` on either side. The flag #4650 introduced is simply set for both variants. Everything that reads what an explode flows follows from that. `FieldValue.ofFieldNamesAndFuseIfPossible` fuses a field access onto an existing field value instead of nesting one inside the other, and the key expression expansion uses it. The pull-up rules require the child of the value being pulled up to be the quantified object value itself, so a nested access could not be pulled through the explode's quantifier at all. An in-join and an in-union bind an element of the collection under `__corr_«alias»`, not the struct the explode flows, so the plans they put underneath have to be translated: `ExplodeExpression.elementBindingTranslationMap` makes each such alias stand for the element alone, which composes `«alias»._0` away to a plain reference to the binding. Neither rule can take an explode with ordinality as a source, since there is no place to put the ordinal. On the SQL side, a quantifier over an unnesting stands for the struct, so the element is reached through `_0`: `LogicalOperator` unwraps it when it builds the unnesting's attributes, `QuantifierValues` sees through it when resolving an index definition down to the base record, and a star expansion over an unnesting expands the element's fields rather than the struct's. The `AT` ordinal is a property of the unnesting rather than a column of the row, so it is ephemeral: nameable, but not part of a star. `ExplodePlanTest` follows the same way: a plain explode flows a record, so the skip-and-limit tests read the element out of it, and the plans that have to flow the element itself ask for that shape explicitly. The hash of a plain explode moves accordingly; the hash a plan that flows the element itself has always had is pinned alongside it.

g31pranjal mentioned this pull request Sep 22, 2026

Flow the WITH ORDINALITY struct as a record constructor value #4651

Merged

g31pranjal added the enhancement New feature or request label Sep 22, 2026

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-value-kind branch from 18b6d61 to 1af4396 Compare September 22, 2026 21:54

g31pranjal requested a review from robert-brunel September 22, 2026 21:55

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-value-kind branch from 1af4396 to fb7c601 Compare September 22, 2026 22:14

g31pranjal requested a review from alecgrieser September 22, 2026 23:05

g31pranjal marked this pull request as ready for review September 22, 2026 23:05

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-value-kind branch from fb7c601 to ef81c06 Compare September 23, 2026 00:01

g31pranjal mentioned this pull request Sep 23, 2026

Flow the array element of an explode wrapped in a struct #4623

Closed

g31pranjal force-pushed the apple/g31pranjal/unnested/explode-value-kind branch from ef81c06 to d46370d Compare September 23, 2026 17:27

robert-brunel requested changes Sep 24, 2026

View reviewed changes

update

ccd9163

g31pranjal requested a review from robert-brunel September 24, 2026 15:36

robert-brunel approved these changes Sep 24, 2026

View reviewed changes

alecgrieser requested changes Sep 25, 2026

View reviewed changes

address comments

91320b1

alecgrieser approved these changes Sep 25, 2026

View reviewed changes

g31pranjal merged commit 73cebd9 into main Sep 25, 2026
14 checks passed

g31pranjal deleted the apple/g31pranjal/unnested/explode-value-kind branch September 25, 2026 13:31

This was referenced Oct 1, 2026

Flow the WITH ORDINALITY struct as a record constructor value #4696

Merged

Stop supporting 1-based explode ordinals #4704

Merged

Conversation

g31pranjal commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why RecordConstructorValue is needed:

Why the shape is recorded rather than inferred

What this PR does

Compatibility

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alecgrieser left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

g31pranjal commented Sep 22, 2026 •

edited

Loading

Why `RecordConstructorValue` is needed: