[g31pranjal]

Registers PushInJoinThroughFetchRule for RecordQueryInComparandJoinPlan, as it already is for RecordQueryInValuesJoinPlan and RecordQueryInParameterJoinPlan. The rule turns InJoin(Fetch(inner)) into Fetch(InJoin(inner)).

SQL IN predicates are planned as in-joins over an InComparandSource, so their plans can now run the in-join over a covering index scan and fetch the records above it. For example, in join-with-order-by-tests.yamsql:

before: [IN arrayDistinct(promote(@c21 AS ARRAY(LONG))) SORTED] | INJOIN q0 -> { ISCAN(t1$a1_a2 [EQUALS q0]) } | FLATMAP q1 -> { ... }
after:  [IN arrayDistinct(promote(@c21 AS ARRAY(LONG))) SORTED] | INJOIN q0 -> { COVERING(t1$a1_a2 [EQUALS q0] -> [a1: KEY:[0], a2: KEY:[1], id: KEY:[2]]) } | FETCH | FLATMAP q1 -> { ... }

Plan changes

• join-with-order-by-tests.yamsql (4 queries) and join-tests-outer.yamsql:460: the in-join runs over a covering index scan, and the records are fetched above it.
• join-tests.yamsql:399: the join order flips, and the in-join over emp now drives the lookup of dept. Both sides are primary-key lookups.
• Planner metrics change in array-join-at, in-predicate, join-tests, join-tests-outer, join-with-order-by-tests, right-deep-plan-tests and versions-tests. No results change.
• PreparedStatementTests.prepareInListOfTuple asserted the order of the rows of a query without ORDER BY, which changes with the new plan. It now checks the set of rows, as Replace ExplodeExpression with a table function over an ExplodeValue #4816 does.

#4816 plans every IN predicate as an in-join over an InExplodeSource, which is a comparand source, and registers this rule too. The changes it lists as not yet accepted (join-tests.yamsql:332, join-tests-outer.yamsql:456 and right-deep-plan-tests.yamsql:37) come from this rule.

[github-actions]

📊 Metrics Diff Analysis Report

Summary

• New queries: 0
• Dropped queries: 0
• Plan changed + metrics changed: 6
• Plan unchanged + metrics changed: 43

ℹ️ About this analysis

This automated analysis compares query planner metrics between the base branch and this PR. It categorizes changes into:

• New queries: Queries added in this PR
• Dropped queries: Queries removed in this PR. These should be reviewed to ensure we are not losing coverage.
• Plan changed + metrics changed: The query plan has changed along with planner metrics.
• Metrics only changed: Same plan but different metrics

The last category in particular may indicate planner regressions that should be investigated.

Plan and Metrics Changed

These queries experienced both plan and metrics changes. This generally indicates that there was some planner change
that means the planning for this query may be substantially different. Some amount of query plan metrics change is expected,
but the reviewer should still validate that these changes are not excessive.

Total: 6 queries

Statistical Summary (Plan and Metrics Changed)

task_count:

• Average change: +155.0
• Average regression: +155.0
• Median change: +164
• Median regression: +164
• Standard deviation: 72.9
• Standard deviation of regressions: 72.9
• Range: +63 to +264
• Range of regressions: +63 to +264
• Queries changed: 6
• Queries regressed: 6

task_count % change distribution (6 queries, bin = 2%):

     Range                                  n
----------  ------------------------------  ---
[+0%, +2%)  ███████████████                 1
[+2%, +4%)  ██████████████████████████████  2
[+4%, +6%)  ██████████████████████████████  2
[+6%, +8%)  ███████████████                 1

transform_count:

• Average change: +19.5
• Average regression: +19.5
• Median change: +21
• Median regression: +21
• Standard deviation: 8.4
• Standard deviation of regressions: 8.4
• Range: +9 to +33
• Range of regressions: +9 to +33
• Queries changed: 6
• Queries regressed: 6

transform_count % change distribution (6 queries, bin = 2%):

     Range                                  n
----------  ------------------------------  ---
[+0%, +2%)  ██████████████████████████████  5
[+2%, +4%)  ██████                          1

transform_yield_count:

• Average change: +4.7
• Average regression: +4.7
• Median change: +5
• Median regression: +5
• Standard deviation: 2.1
• Standard deviation of regressions: 2.1
• Range: +2 to +8
• Range of regressions: +2 to +8
• Queries changed: 6
• Queries regressed: 6

transform_yield_count % change distribution (6 queries, bin = 2%):

     Range                                  n
----------  ------------------------------  ---
[+0%, +2%)  ██████████████████████████████  3
[+2%, +4%)  ██████████████████████████████  3

insert_new_count:

• Average change: +18.7
• Average regression: +18.7
• Median change: +16
• Median regression: +16
• Standard deviation: 8.5
• Standard deviation of regressions: 8.5
• Range: +8 to +32
• Range of regressions: +8 to +32
• Queries changed: 6
• Queries regressed: 6

insert_new_count % change distribution (6 queries, bin = 2%):

     Range                                  n
----------  ------------------------------  ---
[+2%, +4%)  ██████████████████████████████  3
[+4%, +6%)  ██████████████████████████████  3

There were no queries with significant regressions detected.

Minor Changes (Plan and Metrics Changed)

In addition, there were 6 queries with minor changes.

Only Metrics Changed

These queries experienced only metrics changes without any plan changes. If these metrics have substantially changed,
then a planner change has been made which affects planner performance but does not correlate with any new outcomes,
which could indicate a regression.

Total: 43 queries

Statistical Summary (Only Metrics Changed)

task_count:

• Average change: +20.0
• Average regression: +20.0
• Median change: +4
• Median regression: +4
• Standard deviation: 48.0
• Standard deviation of regressions: 48.0
• Range: +2 to +264
• Range of regressions: +2 to +264
• Queries changed: 43
• Queries regressed: 43

task_count % change distribution (43 queries, bin = 2%):

     Range                                  n
----------  ------------------------------  ---
[+0%, +2%)  ██████████████████████████████  40
[+2%, +4%)  ██                              3

transform_count:

• Average change: +13.6
• Average regression: +13.6
• Median change: +12
• Median regression: +12
• Standard deviation: 6.0
• Standard deviation of regressions: 6.0
• Range: +8 to +24
• Range of regressions: +8 to +24
• Queries changed: 5
• Queries regressed: 5

transform_count % change distribution (5 queries, bin = 2%):

     Range                                  n
----------  ------------------------------  ---
[+0%, +2%)  ██████████████████████████████  5

transform_yield_count:

• Average change: +3.4
• Average regression: +3.4
• Median change: +3
• Median regression: +3
• Standard deviation: 1.5
• Standard deviation of regressions: 1.5
• Range: +2 to +6
• Range of regressions: +2 to +6
• Queries changed: 5
• Queries regressed: 5

transform_yield_count % change distribution (5 queries, bin = 2%):

     Range                                  n
----------  ------------------------------  ---
[+0%, +2%)  ██████████████████████████████  5

insert_new_count:

• Average change: +18.2
• Average regression: +18.2
• Median change: +14
• Median regression: +14
• Standard deviation: 12.8
• Standard deviation of regressions: 12.8
• Range: +6 to +43
• Range of regressions: +6 to +43
• Queries changed: 5
• Queries regressed: 5

insert_new_count % change distribution (5 queries, bin = 2%):

     Range                                  n
----------  ------------------------------  ---
[+0%, +2%)  ██████████████████████████████  5

Significant Regressions (Only Metrics Changed)

There were 3 outliers detected. Outlier queries have a significant regression in at least one field. Statistically, this represents either an increase of more than two standard deviations above the mean or a large absolute increase (e.g., 100).

• yaml-tests/src/test/resources/join-tests.metrics.yaml:186: EXPLAIN select dept.id, dept.name, project.name from emp, dept, project where emp.dept_id = dept.id and project.emp_id = emp.id and dept.id in (1, 2)
  • explain: [IN arrayDistinct(promote(@c39 AS ARRAY(LONG)))] | INJOIN q0 -> { SCAN([IS DEPT, EQUALS q0]) | FLATMAP q1 -> { SCAN([IS PROJECT]) | FLATMAP q2 -> { COVERING(EMPDEPT [EQUALS q1.ID] -> [DEPT_ID: KEY:[0], ID: KEY:[2]]) | FILTER q2.EMP_ID EQUALS _.ID | MAP (1 AS _0) AS q3 RETURN q2 } AS q2 RETURN (q1.ID AS ID, q1.NAME AS _1, q2.NAME AS _2) } }
  • task_count: 21936 -> 22200 (+264)
  • transform_count: 8112 -> 8136 (+24)
  • transform_yield_count: 1209 -> 1215 (+6)
  • insert_new_count: 3823 -> 3866 (+43)
• yaml-tests/src/test/resources/join-tests.metrics.yaml:203: EXPLAIN select emp.id, fname from emp, dept where emp.dept_id = dept.id and dept.id in (1, 2) and emp.id not in (1, 5)
  • explain: [IN arrayDistinct(promote(@c23 AS ARRAY(LONG)))] | INJOIN q0 -> { SCAN([IS DEPT, EQUALS q0]) } | FLATMAP q1 -> { COVERING(EMPDEPT [EQUALS q1.ID] -> [DEPT_ID: KEY:[0], ID: KEY:[2]]) | FILTER NOT _.ID IN promote(@c34 AS ARRAY(LONG)) | FETCH AS q2 RETURN (q2.ID AS ID, q2.FNAME AS FNAME) }
  • task_count: 6257 -> 6400 (+143)
  • transform_count: 1873 -> 1889 (+16)
  • transform_yield_count: 368 -> 372 (+4)
  • insert_new_count: 1096 -> 1112 (+16)
• yaml-tests/src/test/resources/join-tests.metrics.yaml:218: EXPLAIN select emp.id, fname from emp, dept where dept.id = emp.dept_id and dept.id in (1, 2) and emp.id not in (1, 5)
  • explain: [IN arrayDistinct(promote(@c23 AS ARRAY(LONG)))] | INJOIN q0 -> { SCAN([IS DEPT, EQUALS q0]) } | FLATMAP q1 -> { COVERING(EMPDEPT <,> -> [DEPT_ID: KEY:[0], ID: KEY:[2]]) | FILTER NOT _.ID IN promote(@c34 AS ARRAY(LONG)) AND q1.ID EQUALS _.DEPT_ID | FETCH AS q2 RETURN (q2.ID AS ID, q2.FNAME AS FNAME) }
  • task_count: 5354 -> 5463 (+109)
  • transform_count: 1622 -> 1634 (+12)
  • transform_yield_count: 319 -> 322 (+3)
  • insert_new_count: 882 -> 894 (+12)

Minor Changes (Only Metrics Changed)

In addition, there were 40 queries with minor changes.

[robert-brunel]

Why did the plan change here? There is no visible FETCH in the plans.

[g31pranjal]

Right. this is sort of an interesting case. At the root there are two shapes:

1. Dept outer, emp in-join inner <- plan in main
2. In-join outer, dept lookup inner <- plan in this PR.

In this PR, Dept outer, emp in-join inner variant also has another contender: Dept outer, covering empdept index with fetch inner. The later wins against the former in LowestOuterCardinalityTiebreaker. This, when compared to 2, loses in LowestNumUnmatchedFieldsTiebreaker.