nearest_ranges takes ties

[marco-mariotti]

Repo pyranges/pyranges1 · base nearest-ranges-strand-direction · head nearest-ties · 6 files, +181 −3

Why

Overlapping intervals are all at distance 0. With the default exclude_overlaps=False,
"the nearest interval" is therefore every interval of other that a query overlaps — one
interval of self covered by 500 intervals of other produces 500 rows, all of them equally
the nearest.

That is rarely what a user asking for "the nearest gene" wants, and on real data the cost is
severe. On 100 million hg38-like intervals against themselves:

behaviour	rows
every tied interval (today)	1,112,177,235
one per query	7,746,427

144x — the difference between fitting in memory and not.

What

gr.nearest_ranges(other)                  # unchanged: every interval at the winning distance
gr.nearest_ranges(other, ties="first")    # one of them, so one row per interval of self

On both PyRanges.nearest_ranges and RangeFrame.nearest_ranges, with the same vocabulary
the rest of the field uses: bedtools spells it closest -t first, GenomicRanges
select="arbitrary", BEDOPS --closest. bioframe, polars-bio and pyranges 0.x report one
interval per query and offer no way to ask for the other behaviour.

• Default unchanged. ties="all" is what nearest_ranges has always done.
• Which interval comes back is unspecified — only that the same input gives the same
  answer. Nothing is sorted to decide it, because the sort would cost more than the option
  saves.
• k still counts distinct distances, not rows: ties="first", k=2 gives two rows, one
  per distance, rather than the combination being rejected.
• An unknown value raises ValueError naming the options, as direction and multiple
  already do, rather than reaching the kernel — a Rust panic is not an Exception, so
  except Exception cannot catch it and the interpreter is aborted.

Where the saving comes from

Not from filtering. Materialising 1.1 billion rows and then keeping one per query costs the
same memory and more time, so the tied rows are never built: the overlap sweep stops at the
first hit per query, each directional sweep emits one row per distinct distance, and the
merge keeps one row per distance bucket. That is ruranges_core::nearest::nearest_with_ties,
so the floor on ruranges rises to 0.1.8.

Version 1.4.2 → 1.4.3.

Tests

Six new tests in tests/unit/test_nearest_ranges.py (10/10 in the file, 128/128 in
tests/unit), plus a doctest on PyRanges.nearest_ranges:

• the property that catches nearly everything — on 400×400 dense pseudo-random intervals,
  over exclude_overlaps ∈ {False, True}, ties="first" answers exactly the queries
  ties="all" answers, at the same distance, with exactly one row each, with an assertion
  that the fixture has ties so it cannot go vacuous;
• one interval covering three: 3 rows → 1;
• a two-sided tie, and overlap-versus-touching so that picking one row cannot pick the wrong
  bucket (a neighbour that merely touches is at distance 1, not 0);
• k=2 reporting one row per distance;
• the directional path, which splits self by strand and searches each half separately, so
  the option has to survive both halves and the recombination;
• an unknown ties on both classes.

Doctests: 462 in pyranges_main and 13 in range_frame pass. ruff format --check and
ruff check clean.

Cross-library

On a tie-rich hand-made pair, ties="first" puts pyranges1 on the same side of the line as
the libraries that have only ever had one behaviour:

library	rows
bioframe 0.8.0 closest	3
polars-bio 0.33.0 nearest	3
pyranges 0.1.4 nearest	3
pyranges1 ties="first"	3
pyranges1 ties="all" (default)	6