Skip to content

synthetic joined record type planning - #4817

Open
g31pranjal wants to merge 1 commit into
mainfrom
apple/g31pranjal/joined-synthetic-table-planning
Open

synthetic joined record type planning#4817
g31pranjal wants to merge 1 commit into
mainfrom
apple/g31pranjal/joined-synthetic-table-planning

Conversation

@g31pranjal

@g31pranjal g31pranjal commented Nov 2, 2026

Copy link
Copy Markdown
Member

No description provided.

@g31pranjal g31pranjal changed the title Apple/g31pranjal/joined synthetic table planning synthetic joined record type planning Nov 2, 2026
@g31pranjal
g31pranjal changed the base branch from main to apple/g31pranjal/joined-record-type-expansion November 2, 2026 00:06
@g31pranjal g31pranjal added the enhancement New feature or request label Nov 2, 2026
@g31pranjal
g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-table-planning branch from 048d012 to e186785 Compare November 2, 2026 10:47
@g31pranjal
g31pranjal changed the base branch from apple/g31pranjal/joined-record-type-expansion to apple/g31pranjal/joined-synthetic-table-constituent-order November 2, 2026 10:47
Base automatically changed from apple/g31pranjal/joined-synthetic-table-constituent-order to main November 2, 2026 10:48
@g31pranjal
g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-table-planning branch from e186785 to 48bdb3d Compare November 2, 2026 10:54
@g31pranjal
g31pranjal changed the base branch from main to apple/g31pranjal/joined-record-type-expansion November 2, 2026 10:54
@g31pranjal
g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-table-planning branch from 48bdb3d to 9493226 Compare November 2, 2026 11:06
@g31pranjal g31pranjal mentioned this pull request Nov 3, 2026
Merged
Base automatically changed from apple/g31pranjal/joined-record-type-expansion to main November 3, 2026 13:44
g31pranjal added a commit that referenced this pull request Nov 3, 2026
@g31pranjal
Describe the planner type of a RecordType, including a synthetic one (
05882f7 
#4829)

`RecordMetaData.getPlannerType` takes record type names and resolves
them with `getRecordType`, which only knows stored record types. So the
planner cannot describe a synthetic record type. In particular,
`IndexExpansionInfo` cannot build the base type of an index defined on
one.

This adds `getPlannerTypeForRecordType(RecordType)` and
`getPlannerTypeForRecordTypes(Collection<RecordType>)`, which describe
the record types they are given rather than looking them up by name. The
name-taking overloads now resolve their names and delegate to them, so
their behaviour is unchanged. `IndexExpansionInfo` passes the indexed
record types it already holds.

Nothing changes observably yet: indexes on synthetic record types are
not offered to the planner. This is groundwork for planning queries
against indexes on joined synthetic record types (#4817).

`RecordMetaDataPlannerTypeTest` covers three cases:

- the new overload describes a joined record type, which the name-taking
overload rejects;
- for stored types, the new overloads agree with the name-taking ones,
including the version pseudo-field when versions are stored;
- several types are described by the union of their fields, and a single
type exactly as on its own.
@g31pranjal
g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-table-planning branch from 9493226 to 472cd8c Compare November 3, 2026 15:24
@g31pranjal g31pranjal mentioned this pull request Nov 4, 2026
Merged
@g31pranjal
g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-table-planning branch from 472cd8c to 833e8cf Compare November 4, 2026 15:43
@g31pranjal
g31pranjal changed the base branch from main to apple/g31pranjal/plan-context-synthetic-indexes November 4, 2026 15:43
g31pranjal added a commit that referenced this pull request Nov 9, 2026
@g31pranjal
Offer indexes on synthetic record types to the planner when a query r…
8ae0d3a 
…eads all their stored constituents (#4840)

`MetaDataPlanContext` never offered indexes on synthetic record types to
the planner. Both `forRecordQuery` and `forRootReference` now also offer
the readable indexes of every synthetic record type whose stored
constituents are all read by the query, found with the new
`RecordMetaData.getSyntheticRecordTypesFromRecordTypes(Collection<RecordType>)`.

`SyntheticRecordType.getStoredRecordTypes()` gives those constituents:
all of a joined type's, only the parent of an unnested type.
Outer-joined constituents are still required for now.

This changes no plans yet: synthetic primary keys are
`ListKeyExpression`s, which index expansion does not support, so such
indexes are skipped. It is groundwork for #4817. Tests are in
`RecordMetaDataBuilderTest`.
Base automatically changed from apple/g31pranjal/plan-context-synthetic-indexes to main November 9, 2026 19:22
@g31pranjal
g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-table-planning branch from 833e8cf to f593028 Compare November 9, 2026 19:29
@g31pranjal
g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-table-planning branch from f593028 to f97cfc9 Compare November 10, 2026 16:23
@github-actions

github-actions Bot commented Nov 10, 2026

Copy link
Copy Markdown

📊 Metrics Diff Analysis Report

Summary

  • New queries: 34
  • Dropped queries: 0
  • Plan changed + metrics changed: 0
  • Plan unchanged + metrics changed: 2
ℹ️ About this analysis

This automated analysis compares query planner metrics between the base branch and this PR. It categorizes changes into:

  • New queries: Queries added in this PR
  • Dropped queries: Queries removed in this PR. These should be reviewed to ensure we are not losing coverage.
  • Plan changed + metrics changed: The query plan has changed along with planner metrics.
  • Metrics only changed: Same plan but different metrics

The last category in particular may indicate planner regressions that should be investigated.

New Queries

Count of new queries by file:

  • yaml-tests/src/test/resources/joined-record-type-indexes-in-larger-joins.metrics.yaml: 2
  • yaml-tests/src/test/resources/joined-record-type-indexes.metrics.yaml: 32

Only Metrics Changed

These queries experienced only metrics changes without any plan changes. If these metrics have substantially changed,
then a planner change has been made which affects planner performance but does not correlate with any new outcomes,
which could indicate a regression.

Total: 2 queries

Statistical Summary (Only Metrics Changed)

task_count:

  • Average change: -27.5
  • Average regression: +120.0
  • Median change: +120
  • Median regression: +120
  • Standard deviation: 147.5
  • Standard deviation of regressions: 0.0
  • Range: -175 to +120
  • Range of regressions: +120 to +120
  • Queries changed: 2
  • Queries regressed: 1

transform_count:

  • Average change: -16.5
  • Average regression: +24.0
  • Median change: +24
  • Median regression: +24
  • Standard deviation: 40.5
  • Standard deviation of regressions: 0.0
  • Range: -57 to +24
  • Range of regressions: +24 to +24
  • Queries changed: 2
  • Queries regressed: 1

transform_yield_count:

  • Average change: -2.0
  • Average regression: +5.0
  • Median change: +5
  • Median regression: +5
  • Standard deviation: 7.0
  • Standard deviation of regressions: 0.0
  • Range: -9 to +5
  • Range of regressions: +5 to +5
  • Queries changed: 2
  • Queries regressed: 1

insert_new_count:

  • Average change: -4.0
  • Average regression: +13.0
  • Median change: +13
  • Median regression: +13
  • Standard deviation: 17.0
  • Standard deviation of regressions: 0.0
  • Range: -21 to +13
  • Range of regressions: +13 to +13
  • Queries changed: 2
  • Queries regressed: 1

insert_reused_count:

  • Average change: +1.0
  • Average regression: +1.0
  • Median change: +1
  • Median regression: +1
  • Standard deviation: 0.0
  • Standard deviation of regressions: 0.0
  • Range: +1 to +1
  • Range of regressions: +1 to +1
  • Queries changed: 1
  • Queries regressed: 1

Significant Regressions (Only Metrics Changed)

There were 2 outliers detected. Outlier queries have a significant regression in at least one field. Statistically, this represents either an increase of more than two standard deviations above the mean or a large absolute increase (e.g., 100).

  • yaml-tests/src/test/resources/aggregate-index-tests.metrics.yaml:1089: EXPLAIN select a, ek.k, b, max(d) from t6, (select k from t6.c where k = 'q') as ek group by a, ek.k, b having a = 1 and max(d) > 100
    • explain: AISCAN(MV20 [EQUALS promote(@c42 AS LONG), EQUALS promote(@c26 AS STRING), [GREATER_THAN promote(@c49 AS LONG)]] BY_GROUP -> [_0: KEY:[0], _1: KEY:[1], _2: KEY:[3], _3: KEY:[2]]) | MAP (_._0 AS A, _._1 AS K, _._2 AS B, _._3 AS _3)
    • task_count: 392 -> 512 (+120)
    • transform_count: 146 -> 170 (+24)
    • transform_yield_count: 45 -> 50 (+5)
    • insert_new_count: 39 -> 52 (+13)
    • insert_reused_count: 1 -> 2 (+1)
  • yaml-tests/src/test/resources/valid-identifiers.metrics.yaml:29: EXPLAIN select t.id from "foo.table$nested.repeated" as t where exists (select * from t."level0.field1" as b, b."level1$field.1" where "level2$array.field.1" = 10)
    • explain: COVERING(foo.table$nested.repeated.idx.field.1.1.1 [EQUALS promote(@c26 AS LONG)] -> [ID: KEY:[2]]) | MAP (_.ID AS ID)
    • task_count: 899 -> 724 (-175)
    • transform_count: 320 -> 263 (-57)
    • transform_yield_count: 69 -> 60 (-9)
    • insert_new_count: 94 -> 73 (-21)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@g31pranjal