You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Build and benchmark a NumPy-native implementation for plain reverse starts and ends integer aggregations. This complements pyjanitor-devs/janitor-rs#70 and the Python/Rust coordination work in #1682.
The implementation should aggregate by candidate ordinal and use right_index only to attach output labels. It must not use original right-row labels as accumulator addresses.
Input contract
For each aggregation call:
arr, starts/ends, and booleans are non-empty and have equal lengths;
let right_len be the number of right candidate ordinals;
starts satisfies 0 <= start < right_len;
ends satisfies 0 < end <= right_len;
invalid direct inputs raise ValueError rather than being silently skipped.
Algorithm
Use compact integer difference arrays.
For starts, the union of all touched suffixes is min(starts):right_len:
The result length is the touched width, not starts.nunique() or ends.nunique(). No seen mask is needed because every valid row touches a non-empty prefix/suffix and the union is contiguous.
Acceptance criteria
Cover sum, prod, min, max, and size where the NumPy formulation preserves each operation’s semantics; document operations that require a different formulation.
Preserve integer dtypes and established overflow behavior, including unsigned dtypes.
Preserve null contributor semantics: null rows touch groups but contribute no value.
Return labels and aggregates in deterministic candidate-ordinal order; ascending original labels are not required.
Add correctness tests for sorted and unsorted right keys, null-filtered sparse labels, overlapping rows, narrow prefixes/suffixes, boundary bounds, all-null contributors, and repeated starts/ends.
Benchmark against the Rust kernels across narrow-width, dense-width, small-input, and large-input cases.
The difference-array approach should reduce range expansion from O(sum(range widths)) to O(number of left rows + touched right width), while allocating only the compact touched domain.
Brief description
Build and benchmark a NumPy-native implementation for plain reverse
startsandendsinteger aggregations. This complements pyjanitor-devs/janitor-rs#70 and the Python/Rust coordination work in #1682.The implementation should aggregate by candidate ordinal and use
right_indexonly to attach output labels. It must not use original right-row labels as accumulator addresses.Input contract
For each aggregation call:
arr,starts/ends, andbooleansare non-empty and have equal lengths;right_lenbe the number of right candidate ordinals;startssatisfies0 <= start < right_len;endssatisfies0 < end <= right_len;ValueErrorrather than being silently skipped.Algorithm
Use compact integer difference arrays.
For
starts, the union of all touched suffixes ismin(starts):right_len:For
ends, the union is0:max(ends):The result length is the touched width, not
starts.nunique()orends.nunique(). Noseenmask is needed because every valid row touches a non-empty prefix/suffix and the union is contiguous.Acceptance criteria
sum,prod,min,max, andsizewhere the NumPy formulation preserves each operation’s semantics; document operations that require a different formulation.Performance hypothesis
The difference-array approach should reduce range expansion from
O(sum(range widths))toO(number of left rows + touched right width), while allocating only the compact touched domain.