Skip to content

[PERF] Update reverse starts-match aggregation boundary - #1684

Open
samukweku wants to merge 14 commits into
devfrom
issue-85-sum-rev-starts-matches
Open

[PERF] Update reverse starts-match aggregation boundary#1684
samukweku wants to merge 14 commits into
devfrom
issue-85-sum-rev-starts-matches

Conversation

@samukweku

@samukweku samukweku commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • update the Python boundary for reverse starts + matches aggregation across sum, min, max, prod, and size
  • omit the redundant length argument for the new janitor-rs API
  • retain a narrow fallback for older installed janitor_rs releases that still require length
  • preserve the existing length argument for unrelated kernels and ordinary reverse paths

Contract

The Rust kernels validate the starts/matches tape contract. start == index.len() is a valid zero-width suffix and consumes no matches entries. Negative starts remain invalid. Empty-input behavior is disclosed in janitor-rs issue #79.

Verification

The focused reverse aggregation suite passes: pixi run pytest tests/functions/test_conditional_join.py -k agg_rev -q (20 passed).

Rust test and boundary changes are tracked in janitor-rs PRs #87, #88, #91, #92, and #93. Full old-vs-compact runtime and memory tables, including tiny through super-large duplicate/unique cases, are documented in janitor-rs issue #85.

Closes #85
Part of #23

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://pyjanitor-devs.github.io/pyjanitor/pr-preview/pr-1684/

Built to branch gh-pages at 2026-08-27 04:22 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@samukweku

Copy link
Copy Markdown
Collaborator Author

Resolved the CI failures from the released janitor_rs compatibility gap.

The pyjanitor test environment currently installs a released Rust extension whose starts+matches sum/min kernels still require length, while the paired PRs remove that redundant argument. The boundary now tries the new API first and retries with index.size only when the old extension reports the missing length argument.

Focused verification: tests/functions/test_conditional_join.py -k 'agg_rev' — 20 passed.

@samukweku samukweku self-assigned this Aug 25, 2026
@samukweku samukweku changed the title [PERF] Update reverse starts-match sum boundary [PERF] Update reverse starts-match aggregation boundary Aug 26, 2026
@samukweku

Copy link
Copy Markdown
Collaborator Author

Strict empty-tape contract

The Rust reverse match kernels intentionally reject an empty matches tape. Pyjanitor therefore must not call them when every candidate range is zero-width and the flattened tape is empty; it short-circuits that case and returns the empty aggregation result instead.

For non-empty tapes, Rust validates the tape shape. Pyjanitor remains responsible for supplying the binary 0/1 match values. This test covers the all-zero-width path so the strict Rust contract is enforced at the integration boundary rather than only documented.

@samukweku

Copy link
Copy Markdown
Collaborator Author

Empty-tape dispatch fix

The strict Rust contract remains intentional: direct reverse match kernels reject an empty matches tape, and reverse positions kernels reject an empty positions tape. Pyjanitor now enforces the integration side of that contract in _get_join_aggs.py: if the candidate tape is empty, it returns the correctly typed empty aggregation frame before dispatching to Rust, even when left_index still contains rows.

ELI5: Python checks whether there are any tickets to hand to Rust. If there are no tickets, it returns an empty answer itself; Rust only receives a real tape that it can validate.

Covered by the existing all-zero-width regression test. Targeted result: 2 passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant