Skip to content

fix: reject non-sentinel negative starts in compare_start_end's width precheck - #52

Merged
samukweku merged 2 commits into
mainfrom
fix-compare-width-precheck-underflow
Aug 24, 2026
Merged

fix: reject non-sentinel negative starts in compare_start_end's width precheck#52
samukweku merged 2 commits into
mainfrom
fix-compare-width-precheck-underflow

Conversation

@samukweku

@samukweku samukweku commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

compare_start_end_*'s expected_matches_width precheck (in src/compare/comp.rs) filtered candidate rows with **s != -1 && **e != -1 && **s < **e before casting both bounds to usize and subtracting. That only rejects the exact -1 sentinel — any other malformed negative start (e.g. -2) that still satisfies start < end in i64 space (starts=[-2], ends=[1]) slipped through, then underflowed (1usize) - ((-2i64) as usize) before ensure_tape_width ever ran: a panic in debug builds, a silently bogus width in release.

Fix (commit 1): require both bounds to already be non-negative (**s >= 0 && **e >= 0) in the wrapper's precheck instead of only excluding -1.

Adversarial review of the first commit found a second, deeper instance of the same root cause (P1): compare_start_end_core itself — not just the wrapper's precheck — has its own independent row-acceptance check, *start == -1 || *end == -1 || *start >= *end, with the identical gap. starts=[-3], ends=[-2] isn't caught by the wrapper's now-fixed precheck's rejection (it correctly contributes 0 to the width sum), but the wrapper still calls into the core regardless, and the core's own check accepts the row (-3 >= -2 is false), casts both bounds to huge-but-still-ordered usize values, and panics indexing right/matches far out of bounds. Reproduced directly against the core (index 18446744073709551613 is out of bounds for array of shape [1]) before the fix.

Fix (commit 2): the same "require non-negative before cast" fix, applied to the core's own check (*start < 0 || *end == -1 || *start >= *end — only start needs the explicit non-negativity check, since start >= 0 && start < end together prove end > 0 too).

Neither fix changes which rows a well-formed caller's start/end pairs produce — both only tighten rejection of malformed input that was already reachable, just not exercised by any prior test.

Test plan

  • cargo test --no-default-features -- 169 passed, 0 failed:
    • non_sentinel_negative_start_does_not_underflow_the_width_precheck -- exercises the wrapper-level fix through the actual compare_start_end_int64 pyfunction via an embedded Python interpreter
    • both_bounds_negative_but_start_less_than_end_contributes_nothing_not_a_panic -- exercises the core-level fix directly against compare_start_end_core with the review's exact counterexample (starts=[-3], ends=[-2]), confirmed to panic before the fix and pass cleanly after
  • cargo clippy --all-targets --all-features -- -D warnings -- clean
  • cargo fmt --check -- clean
  • Built the wheel via maturin develop --release and ran the review's exact counterexample end-to-end through compare_start_end_int64 in Python: returns ([0], [0], 0) cleanly instead of panicking

samukweku and others added 2 commits August 24, 2026 17:48
… precheck

compare_start_end_*'s expected_matches_width precheck filtered rows
with `**s != -1 && **e != -1 && **s < **e` before casting both bounds
to usize and subtracting. That only rejects the exact -1 sentinel --
any other malformed negative start (e.g. -2) that still satisfies
`start < end` in i64 space (starts=[-2], ends=[1]) slipped through,
then underflowed `(1usize) - ((-2i64) as usize)` before
ensure_tape_width ever ran: a panic in debug builds, a silently bogus
width in release.

Requiring both bounds to already be non-negative (`**s >= 0 && **e >=
0`) closes the gap without changing which rows compare_start_end_core
itself treats as contributing candidates: the core's own cast-then-
range-index already naturally produces zero ticks for a row like this
(the resulting usize range is inverted, which Rust's Range iterator
treats as empty rather than panicking), so tightening the precheck to
agree with that doesn't reject anything a real caller would send.

Found during review of #51 (issue #24's comparison-operator PR); not
introduced there -- this precheck predates it. Same bug class as #38
(caller-supplied bounds needing validation before an unchecked cast),
folded in as a standalone fix once #45 (which addressed #38's other
instances) had already merged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
The previous commit fixed the *wrapper*'s width precheck (a separate
computation used only for ensure_tape_width), but compare_start_end_core
has its own independent row-acceptance check that still let a
malformed row through: `*start == -1 || *end == -1 || *start >= *end`
only rejects the exact -1 sentinel, not any negative value.

starts=[-3], ends=[-2]: -3 isn't -1, -2 isn't -1, and -3 >= -2 is
false (-3 < -2), so the row is accepted. Both bounds then cast to
i64::MAX-adjacent usize values that are *still correctly ordered*
relative to each other (start_ < end_ survives the cast, since
twos-complement wraparound preserves relative order for two negative
numbers of similar magnitude), so the loop walks right/matches at an
enormous offset instead of recognizing the range as invalid --
confirmed with a reproduction that panics with "index
18446744073709551613 is out of bounds for array of shape [1]" before
this fix.

Requiring `start >= 0` (replacing the `start == -1`-only check) closes
this: once a row is accepted, start >= 0 and start < end together
prove end > 0 too, so no separate end < 0 check is needed to reach
the same guarantee the previous commit's wrapper-level fix already
established independently.

Found by adversarial review of this PR (P1).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V3KAbkp6JV96KYXNc6EJmN
@samukweku
samukweku force-pushed the fix-compare-width-precheck-underflow branch from 2175ce9 to 6ea0aaa Compare August 24, 2026 07:49
@samukweku
samukweku merged commit 41687fc into main Aug 24, 2026
17 checks passed
samukweku added a commit that referenced this pull request Aug 24, 2026
compare_start_end_core validated negative and inverted start/end ranges
(PR #52) but never checked a positive `end` against `right.len()`, so a
call with `ends[i] > right.len()` walked `right[nn]` out of bounds and
panicked instead of raising a normal ValueError -- reported in issue #53.

Adds the missing `*end as usize > right_len` check, in both the core loop
and the pyfunction wrapper's matches-tape width precheck (which must
reject the same rows, or it demands a needlessly large tape for rows the
core now silently skips).

Deliberately kept as a plain comparison rather than routed through the
crate's existing `checked_range` helper: benchmarking `compare_start_end`
(a hot, per-row comparison kernel) showed checked_range's Option-returning
usize::try_from calls cost ~25-30% more wall time here than the equivalent
plain comparisons, consistent with compare::op's own doc comment on
indirection being measurably costly in this same hot path. No regression
in the final version (benches/kernels.rs's compare_start_end group, n=100
and n=100,000, before vs after).

Adds regression tests for end == right.len() (valid boundary), end ==
right.len() + 1 (issue #53's exact repro, at both the core and Python
wrapper level), and confirms via a built wheel that the previously-panicking
call now returns cleanly.
samukweku added a commit that referenced this pull request Aug 24, 2026
Part of #56: compare_start_end_core (comp.rs) is not the only file in
src/compare/ with unguarded start/end range indexing -- 13 sibling files
share the pattern, none with any bound validation at all (not even the
negative-bound check #52 added to comp.rs). This fixes the first 5:

- comp_ends.rs / comp_starts.rs ("matches already exist"): these take a
  matches/counts tape already sized by the caller, so an invalid row can't
  be silently skipped the way comp.rs skips its own self-owned tape --
  doing so would desynchronize the shared tape index for every row after
  it. Reject the whole call with a PyValueError up front instead.

- comp_first_ends.rs / comp_first_starts.rs / comp_first.rs ("matches
  does not exist yet"): these own their output array end to end, so they
  follow comp.rs's precedent exactly -- an invalid row silently
  contributes zero ticks, computed consistently between the output-size
  precomputation and the main loop.

comp_first_starts.rs needed the least: its start_..end loop was already
safe for a bad start (an empty Range, not a panic) since end is always
right.len(); only the length-sum precomputation (end - start_) could
underflow.

All bound checks are plain comparisons, not the crate's checked_range
helper -- PR #55 measured checked_range costing ~25-30% more wall time
than plain comparisons in this same per-row-comparison hot path, so that
lesson is applied here preemptively rather than rediscovered per file.
No dedicated benchmark exists for these 5 (unlike comp.rs, they have no
extracted _core function benches/kernels.rs can call without a Python
interpreter), so this is a qualitative call, not a measured one.

12 new tests (Python-wrapper level, matching each file's exported
function directly), covering: the out-of-bounds repro, a negative-bound
variant, and the valid boundary (start/end == right.len()) staying
accepted. Verified via a built wheel that every one of the 5 previously-
panicking calls now returns cleanly.

Remaining files under #56: comp_ne.rs and its 5 siblings (comp_ne_1st,
comp_ne_ends, comp_ne_ends_1st, comp_ne_starts, comp_ne_starts_1st), plus
comp_posns.rs/comp_posns_ne.rs (double indirection through `positions`,
needs its own look at what bound `nn`/`indexer` should each check against).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant