Skip to content

quantile(v, p; sorted=true) is O(n) due to unconditional NaN scan #203

Description

@Beforerr

quantile(v, p; sorted=true) scales linearly in length(v) despite the sort being skipped.

using Chairmarks, Statistics
for n in (10_000, 100_000, 1_000_000)
    v = sort(rand(n))
    @info "n=$n" (@b quantile($v, 0.25; sorted=true))
end
# n=10000     2.9 μs
# n=100000   28.5 μs
# n=1000000  300  μs   ≈ time of any(isnan, v) alone

Cause — _quantilesort! (src/Statistics.jl:1038):

if (sorted && (ismissing(v[end]) || (v[end] isa Number && isnan(v[end])))) ||
    any(x -> ismissing(x) || (x isa Number && isnan(x)), v)
    throw(ArgumentError(...))
end

Shape (sorted && check_end) || any(check_all): when sorted=true and v[end] is finite, falls through to the full any scan. The v[end] fast-path never fires on the happy path.

Two options:

  • Trust the contract (sorted ? check_end : any(check_all)) — O(1) when sorted, matches docstring ("can be assumed to be sorted"). Requires updating the test at test/runtests.jl:707-713 which plants NaN mid-array under sorted=true.
  • Drop the dead v[end] branch — preserves current behavior, no perf change, just clarity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions