Skip to content

[PERF] Optimize conditional_join range min/max aggregations - #1674

Open
samukweku wants to merge 6 commits into
devfrom
1653-optimize-range-min-max
Open

[PERF] Optimize conditional_join range min/max aggregations#1674
samukweku wants to merge 6 commits into
devfrom
1653-optimize-range-min-max

Conversation

@samukweku

@samukweku samukweku commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Replace the O(sum of interval widths) Rust compute_min_start*/compute_min_end*/compute_max_start*/compute_max_end* scans with an O(n) NumPy prefix/suffix running-argmin/argmax (_prefix_argext/_suffix_argext in _agg_functions.py), used once query density crosses a benchmarked work-factor threshold (_use_argext); sparse queries keep using the Rust kernels directly, matching the pattern already established in [PERF] Use prefix sums for conditional_join range aggregations (integer dtypes) #1648.
  • Preserves exact first-occurrence tie-breaking, null-skipping, and the existing float NaN-comparison quirk: in the current Rust kernels, current < base_val is IEEE754-false whenever either side is NaN, so a NaN only "freezes" a range's result if it happens to be that specific row's own first non-null value — a NaN found later in the same row's range is silently skipped, same as a null. Prefix scans (always starting at index 0) and suffix scans (each row restarts at its own start) needed different handling of this, since a single shared backward scan is provably insufficient for the suffix case (verified by hand and by property test).
  • Along the way, found and fixed two real bugs in my own draft implementation before it ever reached the module: a uint64int64 cast that silently overflowed for values above i64::MAX, and a sentinel-value scheme that collided with legitimate data at narrow-dtype extremes (e.g. uint8's 255). Caught both via cross-validation against the actual compiled Rust kernels across thousands of randomized dtype-extreme-biased trials before wiring anything in.
  • Scope: forward min/max only (_min_starts/_min_ends/_max_starts/_max_ends), matching the issue's own scope boundary. Arbitrary-interval min/max (_min_starts_ends/_max_starts_ends, would need a sparse table) and the ragged/candidate-mask variants are left untouched, as is the reverse-aggregation path (tracked separately in pyjanitor-devs/janitor-rs#23).

Side finding (not fixed here)

The existing Rust kernel panics (ndarray: index out of bounds) if called with start == n (reading arr[start_] unconditionally, out of bounds). This is pre-existing, unrelated to this change, and this PR's NumPy path handles that input correctly (returns -1, an empty range) — filed as pyjanitor-devs/janitor-rs#27 for a janitor-rs-side fix, since it needs a Rust-side bounds check.

Benchmarks

Isolated A/B within this branch (_use_argext monkeypatched to force the old Rust path vs. the real gated path), output parity verified with pd.testing.assert_frame_equal before timing:

left rows right rows key spread Rust (old) NumPy (new) speedup
500 2,000 50 (dense) 1.805 ms 0.695 ms 2.60x
5,000 20,000 500 (dense) 126.480 ms 1.484 ms 85.20x
20,000 50,000 2,000 (dense) 1265.590 ms 3.287 ms 385.03x
500 2,000 4,000 (sparser) 1.821 ms 0.681 ms 2.67x
50 200 50 (tiny) 0.582 ms 0.618 ms 0.94x (gate correctly keeps this on Rust)

The tiny case confirms the density gate is working as intended — NumPy's O(n) precompute isn't worth it below the threshold, and the gate keeps those calls on the faster Rust path.

Test plan

  • New tests/functions/test_conditional_join_agg_min_max.py: dense-path parity against a naive Rust-loop reference (all 10 dtypes, both min/max), all-null/empty-range, tie-breaking, the NaN-freeze quirk (both prefix and suffix directions), sparse-stays-Rust / dense-uses-numpy gating, and a hypothesis property test (300 examples) against the naive reference.
  • pytest tests/functions/test_conditional_join.py — 241 passed, 1 skipped (pre-existing, unrelated); no regressions.
  • pre-commit run --all-files on changed files (ruff check/format, pydoclint, interrogate) — all pass.
  • Correctness cross-validated directly against the compiled janitor_rs kernels (not just the naive Python reference) across thousands of randomized trials.

Issue #1653

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN

Replace the O(sum of interval widths) Rust min/max scans with an O(n)
NumPy prefix/suffix running-argmin/argmax (O(1) per query after the
precompute), used once query density crosses a benchmarked work-factor
threshold; sparse queries keep using the Rust kernels. Preserves exact
first-occurrence tie-breaking, null-skipping, and the existing float
NaN-comparison quirk (a NaN only freezes a range's result when it's
that row's own first non-null value).

Issue #1653 @samukweku

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
@github-actions

github-actions Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://pyjanitor-devs.github.io/pyjanitor/pr-preview/pr-1674/

Built to branch gh-pages at 2026-08-22 08:03 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

…o min/max dispatch

Consolidate the pixi-run/markdownlint/notebook-conversion reminders (each
was repeated 3-5x across Core Principles, Anti-Patterns, and Learned
Patterns) down to their one dedicated section each. Record the
ELI5-comments preference as a Learned Pattern, and apply it to the new
_min_starts/_min_ends/_max_starts/_max_ends dispatch logic from the
previous commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
@samukweku samukweku self-assigned this Aug 22, 2026
@samukweku
samukweku requested a review from ericmjl August 22, 2026 04:29
samukweku and others added 4 commits August 22, 2026 17:05
Comment-only change: explain the running-argmin/argmax accumulate trick,
the compact-array/count-lookup mapping, the next_valid backward-fill,
and the total_width math line by line, on top of the existing
docstring-level ELI5 explanations.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
np.cumsum's default integer accumulator is platform-dependent (int32 on
64-bit Windows), which could silently overflow for arrays past ~2.1
billion elements. Pin dtype=np.int64 explicitly in both the prefix and
suffix valid-count accumulators.

Found by code review of PR #1674.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
Collapse the near-identical dtype-gate + total_width + fallback block
duplicated across _min_starts/_min_ends/_max_starts/_max_ends into one
shared helper. Also removes _ARGEXT_DTYPE_NAMES, a fifth hand-maintained
copy of the supported-dtype list -- eligibility is now derived directly
from each function's own Rust `mapping` dict, so there's nothing left to
drift out of sync.

Re-benchmarked after the refactor: speedups unchanged within noise
(2.4x-372x on overlapping ranges, gate still correctly favors Rust on
tiny inputs).

Found by code review of PR #1674.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant