[PERF] Optimize conditional_join range min/max aggregations - #1674
Open
samukweku wants to merge 6 commits into
Open
[PERF] Optimize conditional_join range min/max aggregations#1674samukweku wants to merge 6 commits into
samukweku wants to merge 6 commits into
Conversation
Replace the O(sum of interval widths) Rust min/max scans with an O(n) NumPy prefix/suffix running-argmin/argmax (O(1) per query after the precompute), used once query density crosses a benchmarked work-factor threshold; sparse queries keep using the Rust kernels. Preserves exact first-occurrence tie-breaking, null-skipping, and the existing float NaN-comparison quirk (a NaN only freezes a range's result when it's that row's own first non-null value). Issue #1653 @samukweku Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
Contributor
|
…o min/max dispatch Consolidate the pixi-run/markdownlint/notebook-conversion reminders (each was repeated 3-5x across Core Principles, Anti-Patterns, and Learned Patterns) down to their one dedicated section each. Record the ELI5-comments preference as a Learned Pattern, and apply it to the new _min_starts/_min_ends/_max_starts/_max_ends dispatch logic from the previous commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
Comment-only change: explain the running-argmin/argmax accumulate trick, the compact-array/count-lookup mapping, the next_valid backward-fill, and the total_width math line by line, on top of the existing docstring-level ELI5 explanations. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
np.cumsum's default integer accumulator is platform-dependent (int32 on 64-bit Windows), which could silently overflow for arrays past ~2.1 billion elements. Pin dtype=np.int64 explicitly in both the prefix and suffix valid-count accumulators. Found by code review of PR #1674. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
Collapse the near-identical dtype-gate + total_width + fallback block duplicated across _min_starts/_min_ends/_max_starts/_max_ends into one shared helper. Also removes _ARGEXT_DTYPE_NAMES, a fifth hand-maintained copy of the supported-dtype list -- eligibility is now derived directly from each function's own Rust `mapping` dict, so there's nothing left to drift out of sync. Re-benchmarked after the refactor: speedups unchanged within noise (2.4x-372x on overlapping ranges, gate still correctly favors Rust on tiny inputs). Found by code review of PR #1674. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
compute_min_start*/compute_min_end*/compute_max_start*/compute_max_end*scans with an O(n) NumPy prefix/suffix running-argmin/argmax (_prefix_argext/_suffix_argextin_agg_functions.py), used once query density crosses a benchmarked work-factor threshold (_use_argext); sparse queries keep using the Rust kernels directly, matching the pattern already established in [PERF] Use prefix sums for conditional_join range aggregations (integer dtypes) #1648.current < base_valis IEEE754-false whenever either side is NaN, so a NaN only "freezes" a range's result if it happens to be that specific row's own first non-null value — a NaN found later in the same row's range is silently skipped, same as a null. Prefix scans (always starting at index 0) and suffix scans (each row restarts at its ownstart) needed different handling of this, since a single shared backward scan is provably insufficient for the suffix case (verified by hand and by property test).uint64→int64cast that silently overflowed for values abovei64::MAX, and a sentinel-value scheme that collided with legitimate data at narrow-dtype extremes (e.g.uint8's 255). Caught both via cross-validation against the actual compiled Rust kernels across thousands of randomized dtype-extreme-biased trials before wiring anything in.min/maxonly (_min_starts/_min_ends/_max_starts/_max_ends), matching the issue's own scope boundary. Arbitrary-intervalmin/max(_min_starts_ends/_max_starts_ends, would need a sparse table) and the ragged/candidate-mask variants are left untouched, as is the reverse-aggregation path (tracked separately inpyjanitor-devs/janitor-rs#23).Side finding (not fixed here)
The existing Rust kernel panics (
ndarray: index out of bounds) if called withstart == n(readingarr[start_]unconditionally, out of bounds). This is pre-existing, unrelated to this change, and this PR's NumPy path handles that input correctly (returns -1, an empty range) — filed aspyjanitor-devs/janitor-rs#27for ajanitor-rs-side fix, since it needs a Rust-side bounds check.Benchmarks
Isolated A/B within this branch (
_use_argextmonkeypatched to force the old Rust path vs. the real gated path), output parity verified withpd.testing.assert_frame_equalbefore timing:The tiny case confirms the density gate is working as intended — NumPy's O(n) precompute isn't worth it below the threshold, and the gate keeps those calls on the faster Rust path.
Test plan
tests/functions/test_conditional_join_agg_min_max.py: dense-path parity against a naive Rust-loop reference (all 10 dtypes, both min/max), all-null/empty-range, tie-breaking, the NaN-freeze quirk (both prefix and suffix directions), sparse-stays-Rust / dense-uses-numpy gating, and ahypothesisproperty test (300 examples) against the naive reference.pytest tests/functions/test_conditional_join.py— 241 passed, 1 skipped (pre-existing, unrelated); no regressions.pre-commit run --all-fileson changed files (ruff check/format, pydoclint, interrogate) — all pass.janitor_rskernels (not just the naive Python reference) across thousands of randomized trials.Issue #1653
Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN