Describe the bug
With the default closest rounding, a negative percent split boundary whose absolute value rounds to 0 loses its sign, so "relative to the end" silently becomes "relative to the start":
train[-1%:] on a 30-example split returns all 30 rows instead of the last 1% (~0 rows, as train[99%:] correctly yields);
train[:-1%] returns an empty slice and raises ValueError: Instruction "train[:-1%]" corresponds to no data! instead of the first 99% (as train[:99%] correctly yields).
The docs define test[:-5%] as "first 95% of test" (ReadInstruction.from_spec docstring) and use train[-80%:] for "last 80%" (loading.mdx), so the positive and negative spellings of the same boundary should agree.
Root cause: _rel_to_abs_instr first rounds the boundary (_pct_to_abs_closest(-1, 30) == int(round(-0.3)) == 0) and then decides end-relativity from the rounded value (if from_ < 0:). When the rounded value is exactly 0, the sign is gone and the boundary anchors to the start. This triggers whenever round(|pct| * n / 100) == 0. The worst case is train[-X%:], which silently returns 100% of the data (e.g. a "validation" split that equals the full training set).
Steps to reproduce the bug
from datasets.arrow_reader import ReadInstruction
name2len = {"train": 30}
for spec in ["train[-1%:]", "train[99%:]", "train[:-1%]", "train[:99%]"]:
abs_ = ReadInstruction.from_spec(spec).to_absolute(name2len)[0]
print(spec, "->", (abs_.from_, abs_.to))
# train[-1%:] -> (0, 30) # expected ~ (30, 30), like train[99%:]
# train[99%:] -> (30, 30)
# train[:-1%] -> (0, 0) # expected ~ (0, 30), like train[:99%]
# train[:99%] -> (0, 30)
Same end to end: on a 30-row CSV, load_dataset("csv", data_files=..., split="train[-1%:]") returns all 30 rows, and split="train[:-1%]" raises ValueError: Instruction "train[:-1%]" corresponds to no data!.
Expected behavior
train[-1%:] should match train[99%:] (~0 rows here) and train[:-1%] should match train[:99%] (all 30 rows here): whether a boundary is end-relative should be decided by the sign written in the spec, not by the sign of the rounded absolute value. I'll open a PR with a fix.
Environment info
datasets 5.0.2.dev0 (main @ 836b82e), pyarrow 25.0.1, Python 3.13.3, macOS
Disclosure: this report was prepared with AI assistance; I reproduced the behavior locally and reviewed every claim.
Describe the bug
With the default
closestrounding, a negative percent split boundary whose absolute value rounds to 0 loses its sign, so "relative to the end" silently becomes "relative to the start":train[-1%:]on a 30-example split returns all 30 rows instead of the last 1% (~0 rows, astrain[99%:]correctly yields);train[:-1%]returns an empty slice and raisesValueError: Instruction "train[:-1%]" corresponds to no data!instead of the first 99% (astrain[:99%]correctly yields).The docs define
test[:-5%]as "first 95% of test" (ReadInstruction.from_specdocstring) and usetrain[-80%:]for "last 80%" (loading.mdx), so the positive and negative spellings of the same boundary should agree.Root cause:
_rel_to_abs_instrfirst rounds the boundary (_pct_to_abs_closest(-1, 30) == int(round(-0.3)) == 0) and then decides end-relativity from the rounded value (if from_ < 0:). When the rounded value is exactly 0, the sign is gone and the boundary anchors to the start. This triggers wheneverround(|pct| * n / 100) == 0. The worst case istrain[-X%:], which silently returns 100% of the data (e.g. a "validation" split that equals the full training set).Steps to reproduce the bug
Same end to end: on a 30-row CSV,
load_dataset("csv", data_files=..., split="train[-1%:]")returns all 30 rows, andsplit="train[:-1%]"raisesValueError: Instruction "train[:-1%]" corresponds to no data!.Expected behavior
train[-1%:]should matchtrain[99%:](~0 rows here) andtrain[:-1%]should matchtrain[:99%](all 30 rows here): whether a boundary is end-relative should be decided by the sign written in the spec, not by the sign of the rounded absolute value. I'll open a PR with a fix.Environment info
datasets5.0.2.dev0 (main@ 836b82e), pyarrow 25.0.1, Python 3.13.3, macOSDisclosure: this report was prepared with AI assistance; I reproduced the behavior locally and reviewed every claim.