With the recent addition of Iceberg 1.11 compatibility to Comet (#4840) we should codify what versions we will run in CI. #4840 has a table of testing after that PR, which I duplicate below:
CI coverage across Spark and Iceberg versions (rows are the Spark version; columns are the Iceberg runtime paired with it):
| Spark \ Iceberg |
1.5.2 |
1.8.1 |
1.9.1 |
1.10.0 |
1.11.0 |
| 3.4 |
Comet suites — every PR |
Iceberg tests — main + run-iceberg-tests |
— |
— |
— |
| 3.5 |
— |
Comet suites — every PR |
Iceberg tests — main + run-iceberg-tests |
Iceberg tests — main + run-iceberg-tests |
— |
| 4.0 |
— |
— |
— |
Comet suites — every PR |
— |
| 4.1 |
— |
— |
— |
— |
Comet suites + Iceberg tests — every PR |
| 4.2 |
— |
— |
— |
Comet suites — every PR |
— |
Comet suites = CometIcebergNativeSuite (and the other Iceberg-touching Comet unit suites) in the pr_build_linux "scans" bucket; the Iceberg runtime is pulled by the active spark-* profile in spark/pom.xml. Runs on every PR across all five Spark profiles.
Iceberg tests = Apache Iceberg's own iceberg-spark / -extensions / -runtime suites run with Comet enabled, via the iceberg_1_8 / iceberg_1_9 / iceberg_1_10 / iceberg_1_11 jobs. 1.11 (Spark 4.1) runs on every PR; the older three run on push-to-main or on a PR labeled run-iceberg-tests.
We maintain diffs for Iceberg 1.8, 1.9, 1.10, and 1.11 so we can run their Iceberg Java test suites. However, it really seems like each subsequent version is a superset of the former.
We should 1) look at what Iceberg's stated support window is, 2) compare to Spark's stated support window, 3) reconcile with Comet's support/compatibility goals in order to come up with a consistent plan.
With the recent addition of Iceberg 1.11 compatibility to Comet (#4840) we should codify what versions we will run in CI. #4840 has a table of testing after that PR, which I duplicate below:
CI coverage across Spark and Iceberg versions (rows are the Spark version; columns are the Iceberg runtime paired with it):
run-iceberg-testsrun-iceberg-testsrun-iceberg-testsComet suites =
CometIcebergNativeSuite(and the other Iceberg-touching Comet unit suites) in thepr_build_linux"scans" bucket; the Iceberg runtime is pulled by the activespark-*profile inspark/pom.xml. Runs on every PR across all five Spark profiles.Iceberg tests = Apache Iceberg's own
iceberg-spark/-extensions/-runtimesuites run with Comet enabled, via theiceberg_1_8/iceberg_1_9/iceberg_1_10/iceberg_1_11jobs. 1.11 (Spark 4.1) runs on every PR; the older three run on push-to-main or on a PR labeledrun-iceberg-tests.We maintain diffs for Iceberg 1.8, 1.9, 1.10, and 1.11 so we can run their Iceberg Java test suites. However, it really seems like each subsequent version is a superset of the former.
We should 1) look at what Iceberg's stated support window is, 2) compare to Spark's stated support window, 3) reconcile with Comet's support/compatibility goals in order to come up with a consistent plan.