[PERF] Extend selective bound selection to keep='all' for range joins - #1667
Draft
samukweku wants to merge 3 commits into
Draft
[PERF] Extend selective bound selection to keep='all' for range joins#1667samukweku wants to merge 3 commits into
samukweku wants to merge 3 commits into
Conversation
…ch reference Preserves an uncommitted edit from earlier session work (added when researching #1659) that hadn't landed on any branch yet.
Removes the keep in ('first', 'last') guard around #1659's selective
le_lt/ge_gt bound selection for range joins, mirroring #1657's identical
extension of #1658's same-direction anchor selection. Matched row
content is unaffected either way; for keep='all' bound choice can still
affect row order (never guaranteed), which the pre-existing pathology
was unbounded rather than mild for, unlike the first/last case.
Issue #1666.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends #1659's selective
le_lt/ge_gtbound selection for range joins(a
</<=predicate combined with a>/>=predicate) tokeep='all',not just
keep='first'/'last'. Removes thekeep in ('first', 'last')guard around
_maybe_select_better_range_boundsin_get_indices_non_equi.py.This mirrors #1657's identical extension of #1658/#1641's same-direction
anchor selection to
keep='all', on the same basis: the matched rowcontent is invariant to bound choice for every
keepmode (same proof#1641/#1658/#1657/#1659 already established and reuse). Only
keep='all'row order can change - and that was never documented or guaranteed to
begin with.
Depends on #1663 (base branch), which is itself stacked on #1658/#1662.
ELI5
conditional_joincan take several</<=/>/>=conditions at once.For range joins (one
<-type plus one>-type condition), when you giveit more than one candidate for either side, it used to just always use
whichever one you happened to list first as the "anchor" - the one it
uses to narrow down candidate rows before checking the rest. If that
first one wasn't very selective (didn't rule out many rows), the join
could get much slower - unboundedly so for
keep='all', since it's notjust picking one match per row like
keep='first'/'last'does.#1659 already fixed this for
keep='first'/'last'by picking the moreselective candidate instead of just the first-supplied one. This PR
applies the exact same fix to
keep='all'. The only thing that canchange is which order the output rows come back in (which was never
promised to be any particular order anyway) - never which rows, or
their values.
Benchmark
Same skewed-data shape as #1659/#1666's issue write-up (one narrow
<candidate, one broad
<-type "distractor" candidate,keep='all'),before vs. after this fix:
The pre-fix pathology for
keep='all'was unbounded (kept growing past30k rows), unlike the mild, plateauing ~2-9x seen for
keep='first'/'last'before #1659. After this fix,keep='all'timings track thesame selective-bound-choice cost as
keep='first'/'last'already do.Test plan
tests/functions/test_conditional_join.py:test_range_join_bound_selection_used_for_keep_all- asserts_maybe_select_better_range_boundsis now invoked (viawraps=) forkeep='all', inverting the old assert-not-called test.test_range_join_le_lt_content_invariant_keep_all(bothjoin_algorithmvalues) - matched row content is order-of-argumentinvariant for
keep='all'(compared sorted).test_range_join_row_order_can_differ_for_keep_all_on_tie- documentsthe accepted trade-off with an empirically-verified repro: row order
can differ between argument orders only when there's an actual tie in
the
ge_gtright column that also isn't already monotonic (aconstant
ge_gtcolumn never triggers a real sort, so neverreproduces this) - content still matches once sorted.
pixi run pytest tests/functions/test_conditional_join.py -n auto --color=no -q- 291 passed, 1 skipped.pixi run pytest --doctest-modules janitor/functions/conditional_join.py- 2 passed.pixi run linton changed files - clean.