feat: [iceberg] support iceberg metadata columns _file, _partition#4752
Draft
parthchandra wants to merge 4 commits into
Draft
feat: [iceberg] support iceberg metadata columns _file, _partition#4752parthchandra wants to merge 4 commits into
parthchandra wants to merge 4 commits into
Conversation
parthchandra
force-pushed
the
iceberg-metadata-columns
branch
2 times, most recently
from
d23bf63 to
ec21656
Compare
parthchandra
changed the title
parthchandra
force-pushed
the
iceberg-metadata-columns
branch
2 times, most recently
from
50fb6b3 to
0e34cf1
Compare
parthchandra
force-pushed
the
iceberg-metadata-columns
branch
from
cf7b59f to
82d7c43
Compare
hsiang-c
reviewed
Jul 9, 2026
| // Data under spec 0 | ||
| spark.sql(s"INSERT INTO $table VALUES (1, 'US', 'A'), (2, 'EU', 'B')") | ||
|
|
||
| // Evolve partition spec: add category field -> spec 1 |
Contributor
There was a problem hiding this comment.
(nit) We might be able to use SparkSQL as well but it is not a must.
ALTER TABLE $table ADD PARTITION FIELD category; ref: https://iceberg.apache.org/docs/nightly/spark-ddl/#alter-table-add-partition-field
hsiang-c
reviewed
Jul 9, 2026
| withFallbackReason(scan, "Comet Scan is not enabled") | ||
|
|
||
| case scan if metadataCols(scan).nonEmpty => | ||
| case scan: FileSourceScanExec if metadataCols(scan).nonEmpty => |
Contributor
There was a problem hiding this comment.
Maybe we can move it under the data source V1 branch?
hsiang-c
reviewed
Jul 9, 2026
|
|
||
| // _pos should be 0-indexed per file | ||
| val result = spark.sql(s"SELECT id, _pos FROM $table ORDER BY id").collect() | ||
| assert(result(0).getLong(1) == 0L) |
Contributor
There was a problem hiding this comment.
Should be all zero according to my local test.
hsiang-c
reviewed
Jul 9, 2026
| val positions = result.map(_.getLong(0)) | ||
| // First file has 0,1,2; second file has 0,1 (or vice versa) | ||
| assert(positions.contains(0L)) | ||
| assert(positions.count(_ == 0L) == 2, "Each file should start _pos at 0") |
Contributor
There was a problem hiding this comment.
I got this from Iceberg Java:
+----+
|_pos|
+----+
|0 |
|0 |
|0 |
|0 |
|0 |
+----+
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Adds support for iceberg metadata columns
Closes #.
Rationale for this change
What changes are included in this PR?
How are these changes tested?