Skip to content

Negative percent split boundary silently anchors to the start when it rounds to 0 (train[-1%:] returns the whole split) #8473

Description

@codeAnqiang-ma

Describe the bug

With the default closest rounding, a negative percent split boundary whose absolute value rounds to 0 loses its sign, so "relative to the end" silently becomes "relative to the start":

  • train[-1%:] on a 30-example split returns all 30 rows instead of the last 1% (~0 rows, as train[99%:] correctly yields);
  • train[:-1%] returns an empty slice and raises ValueError: Instruction "train[:-1%]" corresponds to no data! instead of the first 99% (as train[:99%] correctly yields).

The docs define test[:-5%] as "first 95% of test" (ReadInstruction.from_spec docstring) and use train[-80%:] for "last 80%" (loading.mdx), so the positive and negative spellings of the same boundary should agree.

Root cause: _rel_to_abs_instr first rounds the boundary (_pct_to_abs_closest(-1, 30) == int(round(-0.3)) == 0) and then decides end-relativity from the rounded value (if from_ < 0:). When the rounded value is exactly 0, the sign is gone and the boundary anchors to the start. This triggers whenever round(|pct| * n / 100) == 0. The worst case is train[-X%:], which silently returns 100% of the data (e.g. a "validation" split that equals the full training set).

Steps to reproduce the bug

from datasets.arrow_reader import ReadInstruction

name2len = {"train": 30}
for spec in ["train[-1%:]", "train[99%:]", "train[:-1%]", "train[:99%]"]:
    abs_ = ReadInstruction.from_spec(spec).to_absolute(name2len)[0]
    print(spec, "->", (abs_.from_, abs_.to))
# train[-1%:] -> (0, 30)   # expected ~ (30, 30), like train[99%:]
# train[99%:] -> (30, 30)
# train[:-1%] -> (0, 0)    # expected ~ (0, 30), like train[:99%]
# train[:99%] -> (0, 30)

Same end to end: on a 30-row CSV, load_dataset("csv", data_files=..., split="train[-1%:]") returns all 30 rows, and split="train[:-1%]" raises ValueError: Instruction "train[:-1%]" corresponds to no data!.

Expected behavior

train[-1%:] should match train[99%:] (~0 rows here) and train[:-1%] should match train[:99%] (all 30 rows here): whether a boundary is end-relative should be decided by the sign written in the spec, not by the sign of the rounded absolute value. I'll open a PR with a fix.

Environment info

  • datasets 5.0.2.dev0 (main @ 836b82e), pyarrow 25.0.1, Python 3.13.3, macOS

Disclosure: this report was prepared with AI assistance; I reproduced the behavior locally and reviewed every claim.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions