Comet 1.0.0 Release

[andygrove]

What is the problem the feature request solves?

I would like to start planning ahead for a 1.0.0 release. Let's use this issue to discuss what that means.

Current planned items:

• Demonstrated cost savings for TPC-H and TPC-DS at SF1000 (1TB)
• Consistent logging
• Thorough documentation for compatibility status
• Review all configuration options and consider renaming some for consistency
• Documented policy for how long we will support Spark versions
• Documented process for preventing major performance regressions
• Documented process for versioning (semver) moving forward. What does that mean for Comet?

Nice to have:

• Implement more aggregate expressions
  • feat: support approx_count_distinct aggregate expression #4819
  • feat: support grouping() and grouping_id() indicator functions #4815
  • feat: support listagg / string_agg aggregate (Spark 4.0+) #4816
  • feat: support max_by and min_by aggregate expressions #4817
  • feat: support Spark HyperLogLog sketch functions (hll_sketch_agg, hll_union_agg, hll_sketch_estimate, hll_union) #4802
  • feat: support approx_percentile / percentile_approx aggregate #4801
  • feat: support native aggregate function mode #4782
  • feat: support PivotFirst aggregate for the optimized PIVOT fast path #4798
  • feat: support kurtosis aggregate #4818
  • feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates #4775
• Implement more window functions

Correctness and critical-priority issues to resolve before 1.0.0

Open issues labeled correctness and/or priority:critical. Items marked 🔴 are also priority:critical. (Generated 2026-07-07)

• 🔴 array_filter: array_compact fast path misfires when IsNotNull operand is not the lambda variable #4830 array_filter: array_compact fast path misfires when IsNotNull operand is not the lambda variable
• 🔴 Distinct-aggregate rewrite can split an incompatible-buffer aggregate across Comet and Spark, causing crash or wrong results #4813 Distinct-aggregate rewrite can split an incompatible-buffer aggregate across Comet and Spark, causing crash or wrong results
• bug: array/map kernels crash on non-null child field via CometLocalTableScanExec #4789 bug: array/map kernels crash on non-null child field via CometLocalTableScanExec
• 🔴 bug: flatten drops nulls and returns wrong results when a sub-array is null #4788 bug: flatten drops nulls and returns wrong results when a sub-array is null
• bug: scalar subquery in RepartitionByExpression crashes with "Subquery N not found" #4787 bug: scalar subquery in RepartitionByExpression crashes with "Subquery N not found"
• 🔴 bug: null IN () returns false instead of null under legacy null-in-empty behavior #4786 bug: null IN () returns false instead of null under legacy null-in-empty behavior
• 🔴 [EPIC] Consistent handling of invalid UTF-8 in native StringType (ingress policy) #4764 [EPIC] Consistent handling of invalid UTF-8 in native StringType (ingress policy)
  • Documented as a known issue in docs: note invalid UTF-8 scan limitation and FFI import risk #4846
• [Bug] map_from_arrays / map_from_entries do not enforce null-key rejection or spark.sql.mapKeyDedupPolicy #4680 [Bug] map_from_arrays / map_from_entries do not enforce null-key rejection or spark.sql.mapKeyDedupPolicy
• [Bug] CAST(BinaryType AS StringType) uses unsafe from_utf8_unchecked (undefined behaviour) #4488 [Bug] CAST(BinaryType AS StringType) uses unsafe from_utf8_unchecked (undefined behaviour)

Describe the potential solution

No response

Additional context

No response

[parthchandra]

I'd like to see iceberg metadata columns support make it #4082 (depends on a bunch of PRs in iceberg-rust though).

[mbutrovich]

I will get Iceberg 1.11 support into shape #4840.