Skip to content

perf: optimize spark_size in spark-expr#4877

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/spark_size-2026-07-09-001
Open

perf: optimize spark_size in spark-expr#4877
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/spark_size-2026-07-09-001

Conversation

@andygrove

Copy link
Copy Markdown
Member

⚠️ Autonomously generated by an LLM. This PR was created end-to-end by an automated agent with no human author. It has passed automated correctness (unit tests + differential fuzz) and micro-benchmark gates, but requires full human review. Do not merge without expert scrutiny.

What changed

Optimizes spark_size in the native spark-expr crate.

Compute list row sizes from the offset buffer instead of allocating a sliced ArrayRef per row via list_array.value(i), removing one heap allocation per row.

Evidence

  • Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs main).
  • Benchmark (criterion, Rust-only): see numbers below; gate required ≥ 5.0% mean improvement with non-overlapping confidence intervals and no regressions.
spark_size_ list of short arrays: 93.333% faster (base 151223ns -> cand 10082ns)
spark_unhex_ valid mixed hex: 19.964% faster (base 125189ns -> cand 100197ns)
spark_unhex_ mixed hex column: 13.986% faster (base 109637ns -> cand 94303ns)
spark_size_ list of long arrays: 94.086% faster (base 156323ns -> cand 9245ns)
spark_unhex_ all valid: 42.128% faster (base 201069ns -> cand 116362ns)
spark_unhex_ with nulls: 36.465% faster (base 179663ns -> cand 114150ns)
spark_unhex_ valid hex: 22.177% faster (base 152680ns -> cand 118821ns)

@andygrove andygrove marked this pull request as ready for review July 10, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant