HIVE-29803: [Iceberg] Support ANCESTORS_OF snapshot lineage command by Aggarwal-Raghav · Pull Request #6681 · apache/hive

Aggarwal-Raghav · 2026-08-06T19:50:23Z

What changes were proposed in this pull request?

Introduce, ANCESTORS_OF command for Iceberg tables, allowing users to inspect the snapshot history and rollback lineage natively in Hive.

Why are the changes needed?

To improve Iceberg metadata visibility. This brings Hive closer to feature parity with Spark, which natively support inspecting Iceberg snapshot history and lineage chains.

Does this PR introduce any user-facing change?

Yes, New DDL

-- Trace lineage from the current snapshot
ALTER TABLE my_iceberg_table EXECUTE ANCESTORS_OF;

-- Trace lineage backwards from a specific historical snapshot
ALTER TABLE my_iceberg_table EXECUTE ANCESTORS_OF(2141260513344612246);

How was this patch tested?

On local setup

Aggarwal-Raghav · 2026-08-06T19:53:14Z

Beeline output

HS2 logs

2026-08-07T01:22:02,573  INFO [HiveServer2-Background-Pool: Thread-102] hive.HiveIcebergSerDe: Using schema from existing table {"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":false,"type":"int"},{"id":2,"name":"data","required":false,"type":"string"}]}
2026-08-07T01:22:02,577  INFO [HiveServer2-Background-Pool: Thread-102] SessionState: +---------------------------+---------------------------+
2026-08-07T01:22:02,577  INFO [HiveServer2-Background-Pool: Thread-102] SessionState: |        snapshot_id        |       timestamp_ms        |
2026-08-07T01:22:02,577  INFO [HiveServer2-Background-Pool: Thread-102] SessionState: +---------------------------+---------------------------+
2026-08-07T01:22:02,577  INFO [HiveServer2-Background-Pool: Thread-102] SessionState: | 2141260513344612246       | 1786040238769             |
2026-08-07T01:22:02,577  INFO [HiveServer2-Background-Pool: Thread-102] SessionState: | 5908440916579273721       | 1786039781897             |
2026-08-07T01:22:02,577  INFO [HiveServer2-Background-Pool: Thread-102] SessionState: | 5008121964096893521       | 1786039777510             |
2026-08-07T01:22:02,577  INFO [HiveServer2-Background-Pool: Thread-102] SessionState: +---------------------------+---------------------------+
2026-08-07T01:22:02,577  INFO [HiveServer2-Background-Pool: Thread-102] metadata.Hive: Dumping metastore api call timing information for : execution phase

Aggarwal-Raghav · 2026-08-13T17:48:14Z

Rebased to resolve merge conflicts

okumin · 2026-08-15T15:09:44Z

@Aggarwal-Raghav
This feature seems useful. I support merging this if there is no alternative. Based on the implementation, I don't think this triggers any side effects. Is that right? If so, this may not belong to the ALTER TABLE family.

Aggarwal-Raghav · 2026-08-15T15:20:17Z

@Aggarwal-Raghav This feature seems useful. I support merging this if there is no alternative. Based on the implementation, I don't think this triggers any side effects. Is that right? If so, this may not belong to the ALTER TABLE family.

thanks a lot @okumin , for replying and you are correct. Its read-only operation, I had a doubt in back of my mind along the same lines but hive exposes iceberg procedures using ALTER hence this current implementation. I'm open to suggestions, maybe as UDF if I can expose is that ok, Just food for thought

SELECT * FROM ancestors_of('tbl_name', snapshotID);

Aggarwal-Raghav · 2026-08-15T20:26:33Z

+            Hive.get(hiveConf).getTable(tableNameStr);
+        Table table = IcebergTableUtil.getTable(hiveConf, hiveTable.getTTable());
+
+        this.serializedTable = SerializationUtil.serializeToBase64(SerializableTable.copyOf(table));


Serialization to Base64 is explicitly done because of this kryo5 error

Caused by: org.apache.hive.com.esotericsoftware.kryo.kryo5.KryoException: java.lang.UnsupportedOperationException: can't get field offset on a hidden class: private final org.apache.iceberg.mr.hive.HiveTableUtil$NonSerializingConfig org.apache.iceberg.mr.hive.HiveTableUtil$$Lambda/0x0000000801d8ec70.arg$1 Serialization trace: hadoopConf (org.apache.iceberg.hadoop.HadoopFileIO) io (org.apache.iceberg.SerializableTable) icebergTable (org.apache.iceberg.mr.hive.udf.GenericUDTFIcebergAncestorsOf) genericUDTF (org.apache.hadoop.hive.ql.plan.UDTFDesc) conf (org.apache.hadoop.hive.ql.exec.UDTFOperator) childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.LimitOperator) reducer (org.apache.hadoop.hive.ql.plan.ReduceWork) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:102) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:71) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:71) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:71) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:71) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.CollectionSerializer.write(CollectionSerializer.java:155) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.CollectionSerializer.write(CollectionSerializer.java:44) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:71) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.CollectionSerializer.write(CollectionSerializer.java:155) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.CollectionSerializer.write(CollectionSerializer.java:44) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:71) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:642) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:71) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.write(FieldSerializer.java:108) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeObject(Kryo.java:627) at org.apache.hadoop.hive.ql.exec.SerializationUtilities.serializeObjectByKryo(SerializationUtilities.java:833) at org.apache.hadoop.hive.ql.exec.SerializationUtilities.serializePlan(SerializationUtilities.java:720) at org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:595) ... 31 more Caused by: java.lang.UnsupportedOperationException: can't get field offset on a hidden class: private final org.apache.iceberg.mr.hive.HiveTableUtil$NonSerializingConfig org.apache.iceberg.mr.hive.HiveTableUtil$$Lambda/0x0000000801d8ec70.arg$1 at jdk.unsupported/sun.misc.Unsafe.objectFieldOffset(Unsafe.java:652) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.UnsafeField.<init>(UnsafeField.java:38) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.CachedFields.newUnsafeField(CachedFields.java:211) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.CachedFields.addField(CachedFields.java:156) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.CachedFields.rebuild(CachedFields.java:99) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.FieldSerializer.<init>(FieldSerializer.java:82) at org.apache.hive.com.esotericsoftware.kryo.kryo5.SerializerFactory$FieldSerializerFactory.newSerializer(SerializerFactory.java:124) at org.apache.hive.com.esotericsoftware.kryo.kryo5.SerializerFactory$FieldSerializerFactory.newSerializer(SerializerFactory.java:108) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.newDefaultSerializer(Kryo.java:469) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.getDefaultSerializer(Kryo.java:454) at org.apache.hive.com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:89) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.getRegistration(Kryo.java:581) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.getRegistration(SerializationUtilities.java:250) at org.apache.hive.com.esotericsoftware.kryo.kryo5.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:112) at org.apache.hive.com.esotericsoftware.kryo.kryo5.Kryo.writeClass(Kryo.java:613) at org.apache.hive.com.esotericsoftware.kryo.kryo5.serializers.ReflectField.write(ReflectField.java:68) ... 63 more

sonarqubecloud · 2026-08-16T07:58:08Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Aggarwal-Raghav · 2026-08-17T17:56:12Z

@okumin, any suggestions on the new approach?

asf-ci-hive added the tests pending label Aug 6, 2026

asf-ci-hive added tests passed and removed tests pending labels Aug 6, 2026

Aggarwal-Raghav force-pushed the HIVE-29803 branch from d0da950 to 93b0d5b Compare August 13, 2026 17:47

asf-ci-hive added tests pending tests passed and removed tests passed tests pending labels Aug 13, 2026

Aggarwal-Raghav force-pushed the HIVE-29803 branch from 93b0d5b to bfd3829 Compare August 15, 2026 20:25

asf-ci-hive added tests pending and removed tests passed labels Aug 15, 2026

Aggarwal-Raghav commented Aug 15, 2026

View reviewed changes

asf-ci-hive added tests unstable and removed tests pending labels Aug 15, 2026

HIVE-29803: [Iceberg] Support ANCESTORS_OF snapshot lineage command

da6e5ed

Aggarwal-Raghav force-pushed the HIVE-29803 branch from bfd3829 to da6e5ed Compare August 16, 2026 06:48

asf-ci-hive added tests pending and removed tests unstable labels Aug 16, 2026

asf-ci-hive added tests passed and removed tests pending labels Aug 16, 2026

Navigation Menu

Uh oh!

HIVE-29803: [Iceberg] Support ANCESTORS_OF snapshot lineage command - #6681

HIVE-29803: [Iceberg] Support ANCESTORS_OF snapshot lineage command#6681
Aggarwal-Raghav wants to merge 1 commit into
apache:masterfrom
Aggarwal-Raghav:HIVE-29803

Aggarwal-Raghav commented Aug 6, 2026

Uh oh!

Aggarwal-Raghav commented Aug 6, 2026 •

edited

Loading

Uh oh!

Aggarwal-Raghav commented Aug 13, 2026

Uh oh!

okumin commented Aug 15, 2026

Uh oh!

Aggarwal-Raghav commented Aug 15, 2026 •

edited

Loading

Uh oh!

Aggarwal-Raghav Aug 15, 2026

Uh oh!

sonarqubecloud Bot commented Aug 16, 2026

Uh oh!

Aggarwal-Raghav commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Aggarwal-Raghav commented Aug 6, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Aggarwal-Raghav commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aggarwal-Raghav commented Aug 13, 2026

Uh oh!

okumin commented Aug 15, 2026

Uh oh!

Aggarwal-Raghav commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aggarwal-Raghav Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Aug 16, 2026

Quality Gate passed

Uh oh!

Aggarwal-Raghav commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Aggarwal-Raghav commented Aug 6, 2026 •

edited

Loading

Aggarwal-Raghav commented Aug 15, 2026 •

edited

Loading