fix: guard unchecked caller-supplied indices across _no_range kernels - #45
Merged
Conversation
samukweku
added a commit
that referenced
this pull request
Aug 24, 2026
- reorder_index initialized its output with Array1::zeros, so a slot rejected by the new checked_index guard was indistinguishable from a legitimate mapping to row 0. pyjanitor's only caller does an unfiltered positional reindex on this output, so that slot would have silently duplicated row 0 into the result. Initialize with the crate's established -1 "no value" sentinel instead. - build_positional_index guarded the read (index[pos]) but left the write (result[n] = val) unguarded against n >= result.len(), where result is sized from the caller-supplied length parameter independent of how many in-bounds entries positions actually yields. Break once n reaches capacity, matching the equivalent guard already used by the sibling index_*_only functions in the same file. - The checked_index(*index_left, arr.len()) guard's ~7-line ELI5 comment was duplicated verbatim across all 6 call sites in the four *_no_range.rs files. Consolidated to one canonical copy (max_rev/max_no_range.rs) with the other 5 sites pointing back to it, per this file's own established "consolidate rather than restate" convention. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S1gZKDRiZoBLZnXXXjW3gt
samukweku
added a commit
that referenced
this pull request
Aug 24, 2026
…nd-skip Adversarial review of PR #45 found two remaining issues in index_builder::reorder_index: - P1: a rejected mapping left the output slot as the crate's usual -1 "no match" sentinel and returned Ok. pyjanitor's only caller does an unfiltered `right.iloc[reordered_positions]` on this output, and pandas treats -1 as the *last* row, not "no match" -- so malformed input silently duplicated a row into the result instead of surfacing as an error. - P2: `starts[bucket] + counts[bucket]` used plain `+=`, which panics on overflow in debug builds and silently wraps in release builds, before the result was ever bounds-checked. reorder_index now returns PyResult and raises ValueError on any unresolvable mapping (out-of-range bucket id, or an overflowing position computed via checked_add instead of +=), rather than ever emitting a -1 into output that's positionally indexed downstream without a filter step. Updates the existing regression test and adds a Learned Patterns entry on when the -1 sentinel is/isn't safe to use.
max_rev/max_no_range.rs, min_rev/min_no_range.rs, sum_rev/sum_no_range.rs, and prod_rev/prod_no_range.rs each indexed arr/booleans by index_left, read straight from the caller-supplied left_index array, with no bound check at all. A negative or too-large index_left panicked (pyo3_runtime.PanicException) instead of failing gracefully. Guards index_left with checked_index, skipping the row when it doesn't resolve; right_index needs no such guard since it's only ever used as a HashMap key, never to index an array. Folds in four more functions found via the same audit (same shape of gap, not previously filed): - compare/comp_no_range.rs and comp_no_range_ne.rs only guarded the -1 sentinel before indexing right/right_booleans by right_pos, never the upper bound -- a positive out-of-range value fell straight through. comp_no_range_ne.rs also gains an ensure_equal_lengths check between right and right_booleans, since both are indexed by the same right_pos and need matching lengths for a single checked_index call to safely cover both. - index_builder::build_positional_index only guarded position < 0, same gap. - index_builder::reorder_index had no guard at all, not even the -1 sentinel, on two chained reads (starts/counts by val, then result by the pos derived from those reads). Measured checked_index's added cost directly (built wheel, timed from Python) across 100/1M/10M rows: compare_no_range, build_positional_index, and reorder_index held flat ~0.5-1.4 ns/row across all three sizes -- genuinely O(1) per element. max_rev_no_range's per-row cost grows with n, but that's pre-existing HashMap-rehashing cost from its dictionary-based grouping, not something this fix introduces (confirmed by the other three, none of which use a HashMap, staying flat). Fixes #38 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S1gZKDRiZoBLZnXXXjW3gt
- reorder_index initialized its output with Array1::zeros, so a slot rejected by the new checked_index guard was indistinguishable from a legitimate mapping to row 0. pyjanitor's only caller does an unfiltered positional reindex on this output, so that slot would have silently duplicated row 0 into the result. Initialize with the crate's established -1 "no value" sentinel instead. - build_positional_index guarded the read (index[pos]) but left the write (result[n] = val) unguarded against n >= result.len(), where result is sized from the caller-supplied length parameter independent of how many in-bounds entries positions actually yields. Break once n reaches capacity, matching the equivalent guard already used by the sibling index_*_only functions in the same file. - The checked_index(*index_left, arr.len()) guard's ~7-line ELI5 comment was duplicated verbatim across all 6 call sites in the four *_no_range.rs files. Consolidated to one canonical copy (max_rev/max_no_range.rs) with the other 5 sites pointing back to it, per this file's own established "consolidate rather than restate" convention. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S1gZKDRiZoBLZnXXXjW3gt
…nd-skip Adversarial review of PR #45 found two remaining issues in index_builder::reorder_index: - P1: a rejected mapping left the output slot as the crate's usual -1 "no match" sentinel and returned Ok. pyjanitor's only caller does an unfiltered `right.iloc[reordered_positions]` on this output, and pandas treats -1 as the *last* row, not "no match" -- so malformed input silently duplicated a row into the result instead of surfacing as an error. - P2: `starts[bucket] + counts[bucket]` used plain `+=`, which panics on overflow in debug builds and silently wraps in release builds, before the result was ever bounds-checked. reorder_index now returns PyResult and raises ValueError on any unresolvable mapping (out-of-range bucket id, or an overflowing position computed via checked_add instead of +=), rather than ever emitting a -1 into output that's positionally indexed downstream without a filter step. Updates the existing regression test and adds a Learned Patterns entry on when the -1 sentinel is/isn't safe to use.
index_starts_and_ends, its _keep_first/_keep_last variants, and build_positional_index_first/last all zipped starts against ends (and counts, where present) with no length check, silently truncating to the shorter array on a mismatch instead of raising -- the same whole-call gap ensure_equal_lengths already closes for the sum/min/max/prod families, and the analogous right/right_booleans check this branch added in comp_no_range_ne.rs. Also drops checked_start_or_none, a byte-for-byte duplicate of checked_end that had crept into the same file, in favor of calling checked_end directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S1gZKDRiZoBLZnXXXjW3gt
index_starts_only(_keep_first/_last), index_ends_only(_keep_first/_last), index_starts_and_ends(_keep_first/_last), and build_positional_index_first/last all collected a validated-range Vec before their tape-width precheck and processing loop ran, even on fully valid input. starts/ends/counts are cheap-to-reiterate ndarray views, so recompute checked_end/checked_range_or_none in each pass instead of paying an O(n) allocation on the hot path; the two build_positional_index_* functions have no width precheck at all, so they drop straight to a single pass with no recomputation. Purely an allocation-strategy change -- same 149/149 tests pass unchanged, same PyValueError/skip/panic behavior on every guarded input. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S1gZKDRiZoBLZnXXXjW3gt
samukweku
force-pushed
the
issue-38-no-range-bounds
branch
from
August 24, 2026 06:43
53de6e5 to
1073e09
Compare
5 tasks
4 tasks
samukweku
added a commit
that referenced
this pull request
Aug 24, 2026
… precheck compare_start_end_*'s expected_matches_width precheck filtered rows with `**s != -1 && **e != -1 && **s < **e` before casting both bounds to usize and subtracting. That only rejects the exact -1 sentinel -- any other malformed negative start (e.g. -2) that still satisfies `start < end` in i64 space (starts=[-2], ends=[1]) slipped through, then underflowed `(1usize) - ((-2i64) as usize)` before ensure_tape_width ever ran: a panic in debug builds, a silently bogus width in release. Requiring both bounds to already be non-negative (`**s >= 0 && **e >= 0`) closes the gap without changing which rows compare_start_end_core itself treats as contributing candidates: the core's own cast-then- range-index already naturally produces zero ticks for a row like this (the resulting usize range is inverted, which Rust's Range iterator treats as empty rather than panicking), so tightening the precheck to agree with that doesn't reject anything a real caller would send. Found during review of #51 (issue #24's comparison-operator PR); not introduced there -- this precheck predates it. Same bug class as #38 (caller-supplied bounds needing validation before an unchecked cast), folded in as a standalone fix once #45 (which addressed #38's other instances) had already merged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
max_rev/max_no_range.rs,min_rev/min_no_range.rs,sum_rev/sum_no_range.rs, andprod_rev/prod_no_range.rseach indexedarr/booleansbyindex_left, read straight from the caller-suppliedleft_indexarray, with no bound check at all. A negative or too-largeindex_leftpanicked (pyo3_runtime.PanicException) instead of failing gracefully.index_leftwithchecked_index, skipping the row when it doesn't resolve;right_indexneeds no such guard since it's only ever used as aHashMapkey, never to index an array.Folded in: four more functions with the same shape of gap
Found while auditing for this specific pattern (
arr[*x as usize]with no/incomplete guard), not previously filed as separate issues -- folded into this PR rather than filed separately per explicit direction:compare/comp_no_range.rsandcomp_no_range_ne.rsonly guarded the-1sentinel before indexingright/right_booleansbyright_pos, never the upper bound -- a positive out-of-range value fell straight through unchecked.comp_no_range_ne.rsalso gains anensure_equal_lengthscheck betweenrightandright_booleans, since both are indexed by the sameright_posand need matching lengths for a singlechecked_indexcall to safely cover both (matching howarr/booleansare validated together elsewhere inaggs/).index_builder::build_positional_indexonly guardedposition < 0, same gap.index_builder::reorder_indexhad no guard at all, not even the-1sentinel, on two chained reads (starts/countsindexed byval, thenresultindexed by theposderived from those reads) -- the most exposed of the four. An adversarial review of the initial sentinel-and-skip fix caught thatreorder_index's sole caller (pyjanitor) does an unfilteredright.iloc[reordered_positions]on the result, and pandas treats-1as the last row, not "no match" -- a malformed mapping would have silently produced a wrong-but-plausible reorderedDataFrameinstead of an error.reorder_indexnow returnsPyResultand raisesValueErroron any unresolvable mapping instead.Follow-up: remaining
index_builder.rsbounds + a length-validation gapTwo more rounds of adversarial review on this same branch turned up further gaps in
index_builder.rsbeyond the four above:index_starts_only/index_ends_only/index_starts_and_endsand their_keep_first/_keep_lastvariants, plusbuild_positional_index_first/_last, gained the samechecked_end/checked_range-style per-row bound checks as the rest of the crate (issue [BUG] _rev/*_no_range.rs indexes arr/booleans by unchecked left_index/right_index values #38 proper).starts/ends/countswere the same length beforezip-ing them, so a mismatch silently truncated to the shortest array instead of raising -- the same shape of gapensure_equal_lengthsalready closes for thesum/min/max/prodfamilies. All five affected functions now callensure_equal_lengthsup front, with regression coverage inaggs::adversarial_bounds_tests::index_builder_starts_ends_functions_reject_mismatched_lengths.Follow-up: drop per-call allocation in the bounds-checked path
Adding the per-row guards above introduced a
Vec<Option<...>>materialization in every one of those nine functions, even on fully valid input (allocate the validated-rangeVec, fill it, sum it for the tape-width precheck, then consume it in the processing loop).starts/ends/countsare cheap-to-reiteratendarrayviews, sochecked_end/checked_range_or_noneare now recomputed in each pass instead -- a cheapO(1)-per-row recompute traded for removing anO(n)heap allocation on this hot join/index-build path. The twobuild_positional_index_*functions have no width precheck at all, so they drop straight to a single pass with no recomputation. Purely an allocation-strategy change; behavior is identical on every guarded input.Rebased onto
mainRebased onto
mainto pick up #46 (thebin_search_*_firstallocation-strategy work), resolving twoAGENTS.mdLearned Patternsconflicts (both were independent appends at the same location, plus one genuine duplicate entry from an earlier merge, which was deduplicated).Performance
Measured
checked_index's added cost directly (built the wheel, timed from Python) across 100 / 1M / 10M rows:compare_no_range,build_positional_index, andreorder_indexheld flat ~0.5-1.4 ns/row across all three sizes -- the guard is genuinelyO(1)per element, not a hiddenO(n)/O(n^2)cost.max_rev_no_range's per-row cost grows withn(15 ns/row at 1M, 50 ns/row at 10M), but that's pre-existingHashMap-rehashing cost from its dictionary-based grouping as distinct keys grow -- not something this fix introduces, confirmed by the other three (noHashMapinvolved) staying flat.Test plan
cargo test --no-default-features-- 167 passed, 0 failedcargo clippy --all-targets --all-features -- -D warnings-- cleancargo fmt --check-- cleancargo bench --no-default-features --no-run-- compilesno_range_and_positional_functions_reject_out_of_bounds_indices_without_panicking) exercises all 8 originally-touched functions with out-of-bounds/mismatched input, confirming each now degrades gracefully (skip, sentinel, orValueError) instead of panickingindex_builder_starts_ends_functions_reject_mismatched_lengths) exercises all 5starts/ends(/counts) functions with mismatched-length inputmaturin develop --releaseand ran the same repros end-to-end plus the tiny/large/very-large timing sweepmainverified clean/mergeable (gh pr view --json mergeable,mergeStateStatus)Fixes #38
🤖 Generated with Claude Code
https://claude.ai/code/session_01S1gZKDRiZoBLZnXXXjW3gt