Apache Iceberg version
main (development)
Please describe the bug 馃悶
Iceberg's V3 format allows at most one live DV per data file.
Thus, a writer replacing a data file's DV must add the new DV and remove the superseeded one in the same snapshot.
The Commit function in RowDelta enforces this rule, but only when the caller also called RemovedDeletes. If a caller adds a replacement DV through AddDeletes and forgets RemoveDeletes, nothing compares the new DV against the DVs already in the current snapshot, and the commit succeeds.
The result is a table with two live DVs on one data file. The commit itself looks clean; the damage appears on the next read, where scan planning rejects the table. can't index multiple deletion vectors for <whatever_the_data_file_path_is>. The table stays unreadable until the new metadata is repaired by hand.
The same gap accepts a delta that adds two DVs for one data file, producing the identical broken state. Currently, Transaction.ReplaceFiles rejects this exact case, so the two write paths currently disagree.
Status
Will be working on a PR with the fix.
Apache Iceberg version
main (development)
Please describe the bug 馃悶
Iceberg's
V3format allows at most one liveDVper data file.Thus, a writer replacing a data file's
DVmust add the newDVand remove the superseeded one in the same snapshot.The
Commitfunction inRowDeltaenforces this rule, but only when the caller also calledRemovedDeletes. If a caller adds a replacement DV throughAddDeletesand forgetsRemoveDeletes, nothing compares the newDVagainst theDVsalready in the current snapshot, and the commit succeeds.The result is a table with two live
DVson one data file. The commit itself looks clean; the damage appears on the next read, where scan planning rejects the table.can't index multiple deletion vectors for <whatever_the_data_file_path_is>. The table stays unreadable until the new metadata is repaired by hand.The same gap accepts a delta that adds two DVs for one data file, producing the identical broken state. Currently,
Transaction.ReplaceFilesrejects this exact case, so the two write paths currently disagree.Status
Will be working on a PR with the fix.