[PERF] NumPy prefix sums for integer conditional_join range-sum aggregations

[samukweku]

Summary

• Adaptively accelerate integer join_agg(..., aggfunc=[(col, "sum")])
  range aggregations. Sparse and moderately sized ranges stay on the existing
  Rust kernels; dense overlapping ranges use one O(n + m) NumPy prefix sum.
• Dispatch using the estimated Rust scan work: NumPy is selected when the sum
  of interval widths exceeds three times the aggregation-array length. Three
  or fewer ranges skip estimation because they cannot cross that threshold.
• Cover the <-only, >-only, and arbitrary [start:end) paths for all eight
  supported integer dtypes.
• Keep float32/float64 on their compensated Rust kernels; float prefix sums are
  tracked separately in [PERF] Use prefix sums for conditional_join range aggregations (float dtypes) #1671.
• Remove the pre-existing duplicate _sum_starts_ends definition.

ELI5

Rust rereads every requested section. NumPy first writes down one running
total for the whole array. We make that extra list only when rereading all the
requested sections would mean walking the whole array more than three times.

Why adaptive dispatch

An unconditional prefix sum regresses selective joins because it scans and
copies the full aggregation array even when only one value is needed. On a
sorted one-million-row right frame with one matching row, adaptive Rust takes
4.07 ms end to end versus 6.17 ms for unconditional prefix construction.

The measured kernel crossover for one million values and 1,000 intervals is:

Width per interval	Total width / n	Selected	Rust	Prefix	Adaptive
1	0.001x	Rust	0.002 ms	1.978 ms	0.006 ms
1,000	1x	Rust	0.643 ms	1.913 ms	0.636 ms
3,000	3x	Rust	1.854 ms	1.873 ms	1.857 ms
4,000	4x	NumPy	2.519 ms	2.035 ms	2.052 ms
10,000	10x	NumPy	6.128 ms	1.996 ms	1.901 ms

For a dense 20,000-by-20,000 < aggregation where every left row matches the
entire right frame, adaptive prefix sums take 0.92 ms end to end versus
254.31 ms when forced through Rust, a 278x speedup.

Correctness contract

• Integer results preserve the Rust kernels' int64 wrapping behavior,
  including uint64 bit reinterpretation.
• Nulls, empty inputs, empty/inverted intervals, and extension-array
  reconstruction retain their existing behavior.
• Float kernels and match/position/reverse kernels are unchanged.

Test plan

• pixi run pytest tests/functions/test_conditional_join_agg_int_sum.py -q
  — 70 passed, covering sparse/dense dispatch and all eight integer dtypes.
• pixi run pytest tests/functions/test_conditional_join_agg_int_sum.py tests/functions/test_conditional_join.py -q
  — 311 passed, 1 skipped.
• 4,800 randomized differential comparisons against the actual Rust
  kernels across all integer dtypes and all three range shapes.
• Ruff check/format, pydoclint, interrogate, and commit hooks pass.

Follow-up

• [PERF] Add adaptive integer range-sum kernels janitor-rs#26 tracks moving the adaptive prefix implementation
  into Rust with a one-buffer dense path and coordinated release.

Closes #1648

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://pyjanitor-devs.github.io/pyjanitor/pr-preview/pr-1673/
Built to branch gh-pages at 2026-08-22 03:07 UTC. Preview will be ready when the GitHub Pages deployment is complete.