Skip to content

Add horizon-aware AUC for anytime evaluation #23

Description

@Chonghe-Jiang

Problem

The current metrics.auc() normalizes over the interval between the first and last recorded submissions. It therefore ignores both:

  • time from the evaluation start to the first valid submission; and
  • time from the last submission to the declared evaluation deadline.

This can make methods with materially different optimization speed receive the same anytime score.

For example, under a declared horizon of [0, 100], these curves currently both return 0.8:

first score 0.8 at t=10, then hold to t=100 -> current AUC 0.8
first score 0.8 at t=90, then hold to t=100 -> current AUC 0.8

With baseline 0 over the full horizon, their scores should instead be 0.72 and 0.08.

Proposed API

Keep the existing auc() behavior for compatibility and add a separate pure metric, for example:

auc_over_horizon(
    curve,
    start=0.0,
    end=budget_seconds,
    baseline=0.0,
    time="t",
    value="best_so_far",
)

Semantics to define

  • use baseline from start until the first valid point;
  • hold the last best-so-far value until end;
  • define no-submission and single-submission behavior;
  • reject invalid horizons such as end <= start;
  • define how points outside the horizon are handled;
  • do not infer whether an early stop was normal or an infrastructure failure—the caller should supply the intended comparison horizon.

Acceptance criteria

  • Tests distinguish early and late first improvements over the same horizon.
  • Tests cover empty, single-point, duplicate-time, boundary, and invalid-horizon inputs.
  • The result is normalized by the declared horizon length.
  • The function remains a pure pandas metric and documents its required columns.
  • Existing auc() callers and behavior remain unchanged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions