Skip to content

fix: [DO NOT MERGE] Validate nested Parquet leaf pruning against datafusion#23391#4866

Draft
mbutrovich wants to merge 1 commit into
apache:mainfrom
mbutrovich:fix_4859
Draft

fix: [DO NOT MERGE] Validate nested Parquet leaf pruning against datafusion#23391#4866
mbutrovich wants to merge 1 commit into
apache:mainfrom
mbutrovich:fix_4859

Conversation

@mbutrovich

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Relates to #4859. (Draft: it validates the fix end to end but should not merge until the DataFusion change is released. See below.)

Rationale for this change

Comet's native Parquet scan read entire nested columns even when Spark's plan showed a pruned nested ReadSchema, so a query needing one leaf of a wide array<struct<...>> fetched and decoded every leaf (#4859 measured ~1.35 TB vs plain Spark's ~30.9 GB for the same files and rows).

The root cause is in DataFusion, not Comet. Spark's SchemaPruning bakes nested pruning into the scan's requiredSchema as a narrower type, and Comet already forwards that pruned schema. DataFusion's projection-mask derivation only pruned nested leaves for get_field expression chains and expanded any other projection (including the whole-column cast the schema adapter inserts for a narrowed nested type) to every leaf of the top-level column. So the pruning was correct in the output but not in the I/O.

The fix is upstream in apache/datafusion#23391, which derives a leaf-level ProjectionMask when a projected column's requested type is a nested subtree of the file's type.

What changes are included in this PR?

No Comet production code changes. Comet already forwards the pruned schema, so the fix is entirely on the DataFusion side.

This draft points the DataFusion dependencies at a fork branch that is DataFusion 54.0.0 plus the fix (via [patch.crates-io] in native/Cargo.toml, with the regenerated native/Cargo.lock), so CI can exercise the change end to end. That patch and lock are temporary and will be replaced by a normal DataFusion version bump once the fix lands in a release, at which point this closes #4859. Not for merge as-is.

It also adds regression tests to CometNativeReaderSuite.

How are these changes tested?

New tests in CometNativeReaderSuite write an array<struct<...>> (and a deeper array<struct<struct<...>>>) where a high-entropy leaf dominates the column size, then assert CometNativeScanExec's bytes_scanned when projecting only the small leaf is a fraction of projecting the large leaf too. Before the fix these were equal (whole column read).

Locally, CometFuzzTestSuite and CometNativeReaderSuite pass against the fork branch, and this PR runs full CI.

@mbutrovich mbutrovich changed the title fix: [DO NOT MERGE] fix: [DO NOT MERGE] Validate nested Parquet leaf pruning against datafusion#23391 Jul 8, 2026
@mbutrovich mbutrovich self-assigned this Jul 8, 2026
@mbutrovich mbutrovich added this to the 1.0.0 milestone Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Comet native Parquet scan reads full nested columns despite pruned Spark ReadSchema

1 participant