Skip to content

fix: widen local table scan child nullability to match native kernels#4843

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:fix-4789-local-scan-nullability
Open

fix: widen local table scan child nullability to match native kernels#4843
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:fix-4789-local-scan-nullability

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #4789.

Rationale for this change

Several Comet array/map expressions crash natively when the input comes through CometLocalTableScanExec with a non-nullable child field. An in-memory Seq[Int] / Map[Int, Int] column encodes its array element / map value as non-null (containsNull=false / valueContainsNull=false), but Comet expression serdes promise nullable children. Feeding a non-null child into a native kernel disagrees with the planned output type:

  • slice: Assertion failed: result_data_type == *expected_type: Function 'spark_array_slice' returned value of type 'List(non-null Int32)' while ... expected: 'List(Int32)'
  • map_entries: ListArray expected data type Struct("key": non-null Int32, "value": Int32) got Struct("key": non-null Int32, "value": non-null Int32)
  • array_insert / SPARK-41233 array prepend: analogous item-vs-array type mismatch

It does not reproduce over a native Parquet scan (which normalizes children to nullable) or with a SQL array(...) / map(...) literal (literal children are nullable), so it is specific to the local scan path.

What changes are included in this PR?

The non-null child is carried in two places, so both must be reconciled:

  1. The Arrow batch produced by CometLocalTableScanExec.mapToReaders (built from originalPlan.schema).
  2. The native ScanExec's declared schema, serialized from op.output in CometSink.convert. scan.rs:build_record_batch casts the imported batch back to this declared schema, so widening only the Arrow data is reverted by the cast.

Changes:

  • Add QueryPlanSerde.widenChildNullability, which recursively widens array elements, map values, and struct fields to nullable while leaving top-level types and map keys unchanged (map keys stay non-null, matching Arrow, and their nullability is never serialized).
  • Add an overridable scanFieldType hook in CometSink (default identity) so the widening is scoped to the local table scan and does not affect Union/Coalesce/shuffle sinks, whose children come from already-normalized native execution.
  • CometLocalTableScanExec overrides the hook and applies the same widening to its Arrow schema, so the declared schema and the batch agree exactly and no per-batch cast is needed.

This mirrors how the Parquet scan path already normalizes children to nullable. No native/Rust changes are required.

How are these changes tested?

Added regression tests that run through the local table scan with ConvertToLocalRelation disabled (so the expression executes natively rather than being folded at plan time):

  • CometArrayExpressionSuite: slice and array_insert on non-null-element arrays.
  • CometMapExpressionSuite: map_entries on a non-null-value map.

All new tests pass, along with the existing CometArrayExpressionSuite, CometMapExpressionSuite, and the LocalTableScan tests in CometExecSuite (including nested NullType in struct/array/map).

An in-memory Seq/Map column read through CometLocalTableScanExec encodes its
array element and map value as non-null (containsNull=false /
valueContainsNull=false), but Comet expression serdes promise nullable
children. Feeding a non-null child into a native array/map kernel fails the
planned-vs-actual type assertion (e.g. spark_array_slice returning
List(non-null Int32) while List(Int32) was promised) or panics building a
ListArray for map_entries.

Widen nested child-field nullability on both sides so they agree: the declared
scan schema (serialized from op.output in CometSink) and the Arrow batch
produced by the local scan. Without widening the declared schema too, the
native ScanExec casts the batch back to the non-null type in build_record_batch.

The widening is scoped to the local table scan via an overridable scanFieldType
hook, leaving other sinks untouched. Map keys stay non-null, matching Arrow.
This mirrors the Parquet scan path, which already reads children as nullable.

Closes apache#4789
@andygrove andygrove force-pushed the fix-4789-local-scan-nullability branch from 095a1cc to 3d0ba56 Compare July 7, 2026 14:59
@andygrove andygrove requested a review from mbutrovich July 7, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: array/map kernels crash on non-null child field via CometLocalTableScanExec

1 participant