Skip to content

feat: Support Spark expression hours_of_time#4860

Open
0lai0 wants to merge 3 commits into
apache:mainfrom
0lai0:feat-3126-expression-HourOfTime
Open

feat: Support Spark expression hours_of_time#4860
0lai0 wants to merge 3 commits into
apache:mainfrom
0lai0:feat-3126-expression-HourOfTime

Conversation

@0lai0

@0lai0 0lai0 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #3126
Supersedes #3615

Rationale for this change

Spark 4.1 introduces TimeType and the hour() function on a TIME child, which resolves through HourExpressionBuilder to HoursOfTime, RuntimeReplaceable whose replacement is StaticInvoke(DateTimeUtils, IntegerType, "getHoursOfTime", Seq(child)).
Without a shim, Comet falls back to the JVM for hour(TIME ...) and for EXTRACT(HOUR FROM TIME ...) (which parses to the same HoursOfTime).

Note that HoursOfTime / TimeType do not exist in Spark 3.4, 3.5, or 4.0 (see the datetime expression audit), so this only needs to be wired in the spark-4.1+ shim.

What changes are included in this PR?

  • spark-4.1+ CometExprShim: match StaticInvoke(DateTimeUtils,"getHoursOfTime", TimeType) and rewrite it to DataFusion's built-in date_part('hour', time). DataFusion's date_part signature already accepts Time64(Nanosecond) and returns Int32, matching Spark's Int result. No new protobuf variant and no new native code path.

  • native planner**: accept LongVal with DataType::Time64(Nanosecond) so Spark 4.1 TIME literals (i64 nanoseconds-since-midnight) deserialize into ScalarValue::Time64Nanosecond, matching Arrow's storage exactly.

How are these changes tested?

  • New SQL file test: spark/src/test/resources/sql-tests/expressions/datetime/hour_of_time.sql
  • ./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite hour_of_time" -Dtest=none

@andygrove

Copy link
Copy Markdown
Member

Thanks @0lai0, but seems like this duplicates part of #4821?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support Spark expression: hours_of_time

2 participants