Skip to content

perf: optimize spark_unhex in spark-expr#4876

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/spark_unhex-2026-07-09-pr1
Open

perf: optimize spark_unhex in spark-expr#4876
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/spark_unhex-2026-07-09-pr1

Conversation

@andygrove

Copy link
Copy Markdown
Member

⚠️ Autonomously generated by an LLM. This PR was created end-to-end by an automated agent with no human author. It has passed automated correctness (unit tests + differential fuzz) and micro-benchmark gates, but requires full human review. Do not merge without expert scrutiny.

What changed

Optimizes spark_unhex in the native spark-expr crate.

Replace unhex's per-digit range-match with a compile-time 256-entry hex lookup table and preallocate the BinaryBuilder to its known output size, cutting per-byte branching and repeated buffer reallocations.

Evidence

  • Correctness: unit tests + seeded differential fuzz (bit-identical Arrow output vs main).
  • Benchmark (criterion, Rust-only): see numbers below; gate required ≥ 5.0% mean improvement with non-overlapping confidence intervals and no regressions.
spark_unhex_ valid mixed hex: 19.964% faster (base 125189ns -> cand 100197ns)
spark_unhex_ mixed hex column: 13.986% faster (base 109637ns -> cand 94303ns)
spark_unhex_ all valid: 42.128% faster (base 201069ns -> cand 116362ns)
spark_unhex_ with nulls: 36.465% faster (base 179663ns -> cand 114150ns)
spark_unhex_ valid hex: 22.177% faster (base 152680ns -> cand 118821ns)

@andygrove andygrove marked this pull request as ready for review July 10, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant