[PERF] Avoid materializing not-equal pairs for keep first or last - #1681
[PERF] Avoid materializing not-equal pairs for keep first or last#1681tunglambk wants to merge 4 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #1681 +/- ##
==========================================
- Coverage 87.56% 85.86% -1.70%
==========================================
Files 95 125 +30
Lines 6819 10018 +3199
==========================================
+ Hits 5971 8602 +2631
- Misses 848 1416 +568 🚀 New features to boost your workflow:
|
14749eb to
287fcf4
Compare
samukweku
left a comment
There was a problem hiding this comment.
thanks @tunglambk
kindly update the docs as well, informing users about this approach for != join
287fcf4 to
d64e55e
Compare
|
Addressed the review in d64e55e: the changelog now credits @tunglambk, and the |
|
Thanks @tunglambk |
|
@tunglambk kindly merge |
PR Description
ELI5
For any left value, the first different right value is either the first right value or, if those are equal, the first value different from it. The same rule works backwards for
keep="last", so the join only needs two candidates rather than a nearly Cartesian list of pairs.Performance
End-to-end timings for 2,000 left and 2,000 right rows (best of three runs):
Correctness
keep="all"remains on the existing code pathValidation
This PR resolves #1651.
PR Checklist
CHANGELOG.mdunder the latest version header.Relevant Reviewers