Skip to content

Flow the WITH ORDINALITY struct as a record constructor value #4696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
g31pranjal merged 3 commits into from
Oct 2, 2026
Merged

Flow the WITH ORDINALITY struct as a record constructor value #4696

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
961da9c
Flow the `WITH ORDINALITY` struct as a record constructor value
g31pranjal Sep 22, 2026
65b6627
changes
g31pranjal Sep 25, 2026
a01f72a
Merge branch 'main' into apple/g31pranjal/unnested/explode-ordinality…
g31pranjal Oct 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions ...re/src/main/java/com/apple/foundationdb/relational/recordlayer/query/LogicalOperator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,10 +311,11 @@
Assert.thatUnchecked(expression.getDataType().getCode() == DataType.Code.ARRAY,
ErrorCode.INVALID_COLUMN_REFERENCE,
() -> String.format(Locale.ROOT, "join correlation can occur only on column of repeated type, not %s type", expression.getDataType()));
final boolean withOrdinality = atAlias.isPresent();

Check notice on line 314 in fdb-relational-core/src/main/java/com/apple/foundationdb/relational/recordlayer/query/LogicalOperator.java

View workflow job for this annotation

GitHub Actions / coverage 

File coverage: 94.9% (282/297 lines) | Changed lines: 100.0% (1/1 lines)
// Ask for 0-based ordinals: `AT` is 1-based per the SQL standard, but the one is added below, where the `AT`
// alias is bound, so that the ordinal an index candidate matches against is the position it stores.
final var explode = new ExplodeExpression(expression.getUnderlying(), withOrdinality, withOrdinality);
// for the case when `withOrdinality` is true, we by default ask for 0-based and with RecordConstructor
// value.
final var explode = new ExplodeExpression(expression.getUnderlying(), withOrdinality, withOrdinality,
withOrdinality);
final var resultingQuantifier = Quantifier.forEach(Reference.initialOf(explode));
final QuantifiedObjectValue flowedObjectValue = resultingQuantifier.getFlowedObjectValue();
final Type flowedObjectType = resultingQuantifier.getFlowedObjectType();
Expand Down
Binary file modified yaml-tests/src/test/resources/array-join-at.metrics.binpb
View file
Open in desktop
Binary file not shown.
22 changes: 11 additions & 11 deletions yaml-tests/src/test/resources/array-join-at.metrics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,8 @@ array-join-at:
WHERE T1."id" IN (1, 2)
ref: array-join-at.yamsql:308
explain: '[IN arrayDistinct(promote(@c21 AS ARRAY(LONG)))] | INJOIN q0 -> { SCAN([IS
T1, EQUALS q0]) } | FLATMAP q1 -> { EXPLODE q1.arr1_nn WITH ORDINALITY AS
q2 RETURN (q1.id AS id, q2._0 AS val, q2._1 + 1 AS at) }'
T1, EQUALS q0]) | FLATMAP q1 -> { EXPLODE q1.arr1_nn WITH ORDINALITY AS q2
RETURN (q1.id AS id, q2._0 AS val, q2._1 + 1 AS at) } }'
task_count: 1162
task_total_time_ms: 14
transform_count: 452
Expand All @@ -196,14 +196,14 @@ array-join-at:
- query: EXPLAIN SELECT "id", "val", "at" FROM T1, T1."arr1_nn" AS "val" AT "at"
WHERE "at" IN (1, 2)
ref: array-join-at.yamsql:320
explain: EXPLODE arrayDistinct(@c19) | FLATMAP q0 -> { SCAN([IS T1]) | FLATMAP
q1 -> { EXPLODE q1.arr1_nn WITH ORDINALITY | FILTER _._1 + 1 EQUALS q0 AS
q2 RETURN (q1.id AS id, q2._0 AS val, q2._1 + 1 AS at) } AS q3 RETURN q3 }
task_count: 1056
task_total_time_ms: 31
transform_count: 421
transform_time_ms: 11
transform_yield_count: 52
explain: SCAN([IS T1]) | FLATMAP q0 -> { EXPLODE q0.arr1_nn WITH ORDINALITY |
FILTER _._1 + 1 IN @c19 AS q1 RETURN (q0.id AS id, q1._0 AS val, q1._1 + 1
AS at) }
task_count: 1045
task_total_time_ms: 27
transform_count: 420
transform_time_ms: 10
transform_yield_count: 51
insert_time_ms: 2
insert_new_count: 115
insert_new_count: 114
insert_reused_count: 6
4 changes: 2 additions & 2 deletions yaml-tests/src/test/resources/array-join-at.yamsql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,7 @@ test_block:
- query: SELECT "id", "val", "at"
FROM T1, T1."arr1_nn" AS "val" AT "at"
WHERE T1."id" IN (1, 2)
- explain: "[IN arrayDistinct(promote(@c21 AS ARRAY(LONG)))] | INJOIN q0 -> { SCAN([IS T1, EQUALS q0]) } | FLATMAP q1 -> { EXPLODE q1.arr1_nn WITH ORDINALITY AS q2 RETURN (q1.id AS id, q2._0 AS val, q2._1 + 1 AS at) }"
- explain: "[IN arrayDistinct(promote(@c21 AS ARRAY(LONG)))] | INJOIN q0 -> { SCAN([IS T1, EQUALS q0]) | FLATMAP q1 -> { EXPLODE q1.arr1_nn WITH ORDINALITY AS q2 RETURN (q1.id AS id, q2._0 AS val, q2._1 + 1 AS at) } }"
- unorderedResult: [
{id: 1, val: 101, at: 1},
{id: 2, val: 201, at: 1},
Expand All @@ -317,7 +317,7 @@ test_block:
- query: SELECT "id", "val", "at"
FROM T1, T1."arr1_nn" AS "val" AT "at"
WHERE "at" IN (1, 2)
- explain: "EXPLODE arrayDistinct(@c19) | FLATMAP q0 -> { SCAN([IS T1]) | FLATMAP q1 -> { EXPLODE q1.arr1_nn WITH ORDINALITY | FILTER _._1 + 1 EQUALS q0 AS q2 RETURN (q1.id AS id, q2._0 AS val, q2._1 + 1 AS at) } AS q3 RETURN q3 }"
- explain: "SCAN([IS T1]) | FLATMAP q0 -> { EXPLODE q0.arr1_nn WITH ORDINALITY | FILTER _._1 + 1 IN @c19 AS q1 RETURN (q0.id AS id, q1._0 AS val, q1._1 + 1 AS at) }"

@alecgrieser alecgrieser Oct 2, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the sake of fairness, this plan is better, effectively because it makes the opposite transformation as: https://github.com/FoundationDB/fdb-record-layer/pull/4696/changes#r4166198242 Which is more evidence that this is random plan hash noise, I guess

- resultMetadata: [{id: BIGINT}, {val: INTEGER}, {at: INTEGER}]
- unorderedResult: [
{id: 1, val: 101, at: 1},
Expand Down
43 changes: 22 additions & 21 deletions yaml-tests/src/test/resources/in-predicate.metrics.binpb
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 19 additions & 19 deletions yaml-tests/src/test/resources/in-predicate.metrics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -488,10 +488,10 @@ unnamed-2:
- query: EXPLAIN SELECT t.id, n, idx FROM array_table AS t, t.numbers AS n AT idx
WHERE n IN (10, 20)
ref: in-predicate.yamsql:385
explain: SCAN([IS ARRAY_TABLE]) | FLATMAP q0 -> { EXPLODE arrayDistinct(promote(@c23
AS ARRAY(LONG))) | FLATMAP q1 -> { EXPLODE q0.NUMBERS WITH ORDINALITY | FILTER
_._0 EQUALS q1 AS q2 RETURN q2 } AS q2 RETURN (q0.ID AS ID, q2._0 AS N, q2._1
+ 1 AS IDX) }
explain: EXPLODE arrayDistinct(promote(@c23 AS ARRAY(LONG))) | FLATMAP q0 -> {
SCAN([IS ARRAY_TABLE]) | FLATMAP q1 -> { EXPLODE q1.NUMBERS WITH ORDINALITY
| FILTER _._0 EQUALS q0 AS q2 RETURN (q1.ID AS ID, q2._0 AS N, q2._1 + 1 AS
IDX) } AS q3 RETURN q3 }
task_count: 1060
task_total_time_ms: 82
transform_count: 423
Expand All @@ -504,23 +504,23 @@ unnamed-2:
WHERE idx IN (1, 3)
ref: in-predicate.yamsql:394
explain: EXPLODE arrayDistinct(@c23) | FLATMAP q0 -> { SCAN([IS ARRAY_TABLE])
| FLATMAP q1 -> { EXPLODE q1.NUMBERS WITH ZERO BASED ORDINALITY | FILTER _._1
+ 1 EQUALS q0 AS q2 RETURN (q1.ID AS ID, q2._0 AS N, q2._1 + 1 AS IDX) } AS
q3 RETURN q3 }
task_count: 1060
task_total_time_ms: 96
transform_count: 423
transform_time_ms: 29
transform_yield_count: 54
insert_time_ms: 4
insert_new_count: 115
| FLATMAP q1 -> { EXPLODE q1.NUMBERS WITH ORDINALITY | FILTER _._1 + 1 EQUALS
q0 AS q2 RETURN (q1.ID AS ID, q2._0 AS N, q2._1 + 1 AS IDX) } AS q3 RETURN
q3 }
task_count: 1049
task_total_time_ms: 71
transform_count: 422
transform_time_ms: 22
transform_yield_count: 53
insert_time_ms: 3
insert_new_count: 114
insert_reused_count: 6
- query: EXPLAIN SELECT t.id, n, idx FROM array_table AS t, t.numbers AS n AT idx
WHERE t.id IN (2, 4)
ref: in-predicate.yamsql:405
explain: '[IN arrayDistinct(promote(@c25 AS ARRAY(LONG)))] | INJOIN q0 -> { SCAN([IS
ARRAY_TABLE, EQUALS q0]) | FLATMAP q1 -> { EXPLODE q1.NUMBERS WITH ORDINALITY
AS q2 RETURN (q1.ID AS ID, q2._0 AS N, q2._1 + 1 AS IDX) } }'
ARRAY_TABLE, EQUALS q0]) } | FLATMAP q1 -> { EXPLODE q1.NUMBERS WITH ORDINALITY
AS q2 RETURN (q1.ID AS ID, q2._0 AS N, q2._1 + 1 AS IDX) }'
task_count: 1166
task_total_time_ms: 104
transform_count: 454
Expand All @@ -532,9 +532,9 @@ unnamed-2:
- query: EXPLAIN SELECT t.id, f, idx FROM array_table AS t, t.fruits AS f AT idx
WHERE f IN ('apple', 'mango')
ref: in-predicate.yamsql:413
explain: SCAN([IS ARRAY_TABLE]) | FLATMAP q0 -> { EXPLODE arrayDistinct(@c23)
| FLATMAP q1 -> { EXPLODE q0.FRUITS WITH ORDINALITY | FILTER _._0 EQUALS q1
AS q2 RETURN q2 } AS q2 RETURN (q0.ID AS ID, q2._0 AS F, q2._1 + 1 AS IDX)
explain: EXPLODE arrayDistinct(@c23) | FLATMAP q0 -> { SCAN([IS ARRAY_TABLE])
| FLATMAP q1 -> { EXPLODE q1.FRUITS WITH ORDINALITY | FILTER _._0 EQUALS q0
AS q2 RETURN (q1.ID AS ID, q2._0 AS F, q2._1 + 1 AS IDX) } AS q3 RETURN q3
}
task_count: 1060
task_total_time_ms: 78
Expand Down
6 changes: 3 additions & 3 deletions yaml-tests/src/test/resources/in-predicate.yamsql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -382,7 +382,7 @@ test_block:
# The ordinal (IDX) reflects the position in the original array, not the rank among filtered rows.
- query: SELECT t.id, n, idx FROM array_table AS t, t.numbers AS n AT idx WHERE n IN (10, 20)
- supported_version: 4.12.5.0
- explain: "SCAN([IS ARRAY_TABLE]) | FLATMAP q0 -> { EXPLODE arrayDistinct(promote(@c23 AS ARRAY(LONG))) | FLATMAP q1 -> { EXPLODE q0.NUMBERS WITH ORDINALITY | FILTER _._0 EQUALS q1 AS q2 RETURN q2 } AS q2 RETURN (q0.ID AS ID, q2._0 AS N, q2._1 + 1 AS IDX) }"
- explain: "EXPLODE arrayDistinct(promote(@c23 AS ARRAY(LONG))) | FLATMAP q0 -> { SCAN([IS ARRAY_TABLE]) | FLATMAP q1 -> { EXPLODE q1.NUMBERS WITH ORDINALITY | FILTER _._0 EQUALS q0 AS q2 RETURN (q1.ID AS ID, q2._0 AS N, q2._1 + 1 AS IDX) } AS q3 RETURN q3 }"

@alecgrieser alecgrieser Oct 2, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This plan is worse, because we're now executing the SCAN([IS ARRAY_TABLE]) operator once per element in @c23 rather than once for the whole plan. I'm a little surprised that this change would cause that though unless this was relying on the plan hash tiebreaker or something.

- unorderedResult: [
{ID: 1, N: 10, IDX: 1}, {ID: 1, N: 20, IDX: 2},
{ID: 3, N: 10, IDX: 1},
Expand All @@ -402,15 +402,15 @@ test_block:
# PartiQL join AT on bigint array with IN predicate on the primary key.
- query: SELECT t.id, n, idx FROM array_table AS t, t.numbers AS n AT idx WHERE t.id IN (2, 4)
- supported_version: 4.12.5.0
- explain: "[IN arrayDistinct(promote(@c25 AS ARRAY(LONG)))] | INJOIN q0 -> { SCAN([IS ARRAY_TABLE, EQUALS q0]) | FLATMAP q1 -> { EXPLODE q1.NUMBERS WITH ORDINALITY AS q2 RETURN (q1.ID AS ID, q2._0 AS N, q2._1 + 1 AS IDX) } }"
- explain: "[IN arrayDistinct(promote(@c25 AS ARRAY(LONG)))] | INJOIN q0 -> { SCAN([IS ARRAY_TABLE, EQUALS q0]) } | FLATMAP q1 -> { EXPLODE q1.NUMBERS WITH ORDINALITY AS q2 RETURN (q1.ID AS ID, q2._0 AS N, q2._1 + 1 AS IDX) }"
- unorderedResult: [
{ID: 2, N: 40, IDX: 1}, {ID: 2, N: 50, IDX: 2},
{ID: 4, N: 80, IDX: 1}]
-
# PartiQL join AT on string array with IN predicate on the unnested value.
- query: SELECT t.id, f, idx FROM array_table AS t, t.fruits AS f AT idx WHERE f IN ('apple', 'mango')
- supported_version: 4.12.5.0
- explain: "SCAN([IS ARRAY_TABLE]) | FLATMAP q0 -> { EXPLODE arrayDistinct(@c23) | FLATMAP q1 -> { EXPLODE q0.FRUITS WITH ORDINALITY | FILTER _._0 EQUALS q1 AS q2 RETURN q2 } AS q2 RETURN (q0.ID AS ID, q2._0 AS F, q2._1 + 1 AS IDX) }"
- explain: "EXPLODE arrayDistinct(@c23) | FLATMAP q0 -> { SCAN([IS ARRAY_TABLE]) | FLATMAP q1 -> { EXPLODE q1.FRUITS WITH ORDINALITY | FILTER _._0 EQUALS q0 AS q2 RETURN (q1.ID AS ID, q2._0 AS F, q2._1 + 1 AS IDX) } AS q3 RETURN q3 }"

@alecgrieser alecgrieser Oct 2, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This plan is also worse, for effectively the same reason.

- unorderedResult: [
{ID: 1, F: 'apple', IDX: 1},
{ID: 3, F: 'apple', IDX: 1}, {ID: 3, F: 'mango', IDX: 3}]
Expand Down
Loading