Skip to content

[PERF] Extend selective bound selection to keep='all' for range joins - #1667

Draft
samukweku wants to merge 3 commits into
issue-1659-range-join-anchorfrom
issue-1666-keep-all-range-bounds
Draft

[PERF] Extend selective bound selection to keep='all' for range joins#1667
samukweku wants to merge 3 commits into
issue-1659-range-join-anchorfrom
issue-1666-keep-all-range-bounds

Conversation

@samukweku

Copy link
Copy Markdown
Collaborator

Summary

Extends #1659's selective le_lt/ge_gt bound selection for range joins
(a </<= predicate combined with a >/>= predicate) to keep='all',
not just keep='first'/'last'. Removes the keep in ('first', 'last')
guard around _maybe_select_better_range_bounds in
_get_indices_non_equi.py.

This mirrors #1657's identical extension of #1658/#1641's same-direction
anchor selection to keep='all', on the same basis: the matched row
content is invariant to bound choice for every keep mode (same proof
#1641/#1658/#1657/#1659 already established and reuse). Only keep='all'
row order can change - and that was never documented or guaranteed to
begin with.

Depends on #1663 (base branch), which is itself stacked on #1658/#1662.

ELI5

conditional_join can take several </<=/>/>= conditions at once.
For range joins (one <-type plus one >-type condition), when you give
it more than one candidate for either side, it used to just always use
whichever one you happened to list first as the "anchor" - the one it
uses to narrow down candidate rows before checking the rest. If that
first one wasn't very selective (didn't rule out many rows), the join
could get much slower - unboundedly so for keep='all', since it's not
just picking one match per row like keep='first'/'last' does.

#1659 already fixed this for keep='first'/'last' by picking the more
selective candidate instead of just the first-supplied one. This PR
applies the exact same fix to keep='all'. The only thing that can
change is which order the output rows come back in (which was never
promised to be any particular order anyway) - never which rows, or
their values.

Benchmark

Same skewed-data shape as #1659/#1666's issue write-up (one narrow <
candidate, one broad <-type "distractor" candidate, keep='all'),
before vs. after this fix:

rows before after
1,000 ~0.005s (2.18x baseline) 0.0027s
3,000 ~0.03s (5.84x) 0.0025s
10,000 ~0.3s (32.05x) 0.0087s
30,000 ~3.5s (154.30x, still growing) 0.0613s

The pre-fix pathology for keep='all' was unbounded (kept growing past
30k rows), unlike the mild, plateauing ~2-9x seen for keep='first'/
'last' before #1659. After this fix, keep='all' timings track the
same selective-bound-choice cost as keep='first'/'last' already do.

Test plan

  • New tests in tests/functions/test_conditional_join.py:
    • test_range_join_bound_selection_used_for_keep_all - asserts
      _maybe_select_better_range_bounds is now invoked (via wraps=) for
      keep='all', inverting the old assert-not-called test.
    • test_range_join_le_lt_content_invariant_keep_all (both
      join_algorithm values) - matched row content is order-of-argument
      invariant for keep='all' (compared sorted).
    • test_range_join_row_order_can_differ_for_keep_all_on_tie - documents
      the accepted trade-off with an empirically-verified repro: row order
      can differ between argument orders only when there's an actual tie in
      the ge_gt right column that also isn't already monotonic (a
      constant ge_gt column never triggers a real sort, so never
      reproduces this) - content still matches once sorted.
  • pixi run pytest tests/functions/test_conditional_join.py -n auto --color=no -q - 291 passed, 1 skipped.
  • pixi run pytest --doctest-modules janitor/functions/conditional_join.py - 2 passed.
  • pixi run lint on changed files - clean.

…ch reference

Preserves an uncommitted edit from earlier session work (added when
researching #1659) that hadn't landed on any branch yet.
Removes the keep in ('first', 'last') guard around #1659's selective
le_lt/ge_gt bound selection for range joins, mirroring #1657's identical
extension of #1658's same-direction anchor selection. Matched row
content is unaffected either way; for keep='all' bound choice can still
affect row order (never guaranteed), which the pre-existing pathology
was unbounded rather than mild for, unlike the first/last case.

Issue #1666.
@samukweku samukweku self-assigned this Aug 21, 2026
@samukweku
samukweku requested a review from ericmjl August 21, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant