Skip to content

perf: remove redundant explicit reverse range length - #1692

Open
samukweku wants to merge 4 commits into
devfrom
issue-107-explicit-ranges
Open

perf: remove redundant explicit reverse range length#1692
samukweku wants to merge 4 commits into
devfrom
issue-107-explicit-ranges

Conversation

@samukweku

Copy link
Copy Markdown
Collaborator

Summary

Closes the pyjanitor side of janitor-rs#107 and contributes to #23.

The Rust explicit reverse starts/ends kernels now derive their own bounded capacity hint and no longer accept a redundant length argument. This change removes the corresponding Python-side ends.max() - starts.min() computation and stops passing length to the size kernel.

ELI5: the Python layer was calculating a size estimate that Rust could calculate more safely from the actual ranges. Removing it keeps one source of truth and avoids reserving based on a potentially misleading span, especially with duplicate labels.

Performance and memory

The paired janitor-rs PR includes Criterion comparisons against the old HashMap-per-label implementation for unique and duplicate labels at n=32, 10,000, 100,000, and 1,000,000. The compact kernels are generally ~1.5–3x faster for min/max/prod/size and ~15–40% faster for sum, with broadly comparable memory and substantially lower peak live memory in representative duplicate-label cases. The benchmark records allocation count, allocated bytes, and peak live memory.

Verification

  • Python syntax checks passed.
  • Pre-commit checks passed except pixi-install, which was skipped because dependency resolution was unavailable due to a transient DNS failure; ruff, formatting, pydoclint, interrogate, whitespace, and file checks passed.

Coordinated janitor-rs PR: branch issue-107-explicit-ranges.

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://pyjanitor-devs.github.io/pyjanitor/pr-preview/pr-1692/

Built to branch gh-pages at 2026-08-27 04:37 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@samukweku

Copy link
Copy Markdown
Collaborator Author

Contract documentation is now included in this PR and its branch AGENTS.md.

  • Direct Rust reverse-match kernels reject an empty flattened matches tape; malformed tape shape is not silently accepted.
  • A batch with no candidates at all is valid at the Python level, so pyjanitor pre-filters that case and returns the normal typed empty result before dispatching to Rust.
  • Individual zero-width ranges remain valid when the overall tape is non-empty.
  • The comparison/producer layer owns the matches values-in-{0,1} invariant; the Rust boundary validates tape shape.
  • The compatibility length argument exists only for older janitor-rs wheels.

An ELI5 explanation is included in the Python module docstring and AGENTS.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant