Problem
The current metrics.auc() normalizes over the interval between the first and last recorded submissions. It therefore ignores both:
- time from the evaluation start to the first valid submission; and
- time from the last submission to the declared evaluation deadline.
This can make methods with materially different optimization speed receive the same anytime score.
For example, under a declared horizon of [0, 100], these curves currently both return 0.8:
first score 0.8 at t=10, then hold to t=100 -> current AUC 0.8
first score 0.8 at t=90, then hold to t=100 -> current AUC 0.8
With baseline 0 over the full horizon, their scores should instead be 0.72 and 0.08.
Proposed API
Keep the existing auc() behavior for compatibility and add a separate pure metric, for example:
auc_over_horizon(
curve,
start=0.0,
end=budget_seconds,
baseline=0.0,
time="t",
value="best_so_far",
)
Semantics to define
- use
baseline from start until the first valid point;
- hold the last best-so-far value until
end;
- define no-submission and single-submission behavior;
- reject invalid horizons such as
end <= start;
- define how points outside the horizon are handled;
- do not infer whether an early stop was normal or an infrastructure failure—the caller should supply the intended comparison horizon.
Acceptance criteria
- Tests distinguish early and late first improvements over the same horizon.
- Tests cover empty, single-point, duplicate-time, boundary, and invalid-horizon inputs.
- The result is normalized by the declared horizon length.
- The function remains a pure pandas metric and documents its required columns.
- Existing
auc() callers and behavior remain unchanged.
Problem
The current
metrics.auc()normalizes over the interval between the first and last recorded submissions. It therefore ignores both:This can make methods with materially different optimization speed receive the same anytime score.
For example, under a declared horizon of
[0, 100], these curves currently both return0.8:With baseline 0 over the full horizon, their scores should instead be 0.72 and 0.08.
Proposed API
Keep the existing
auc()behavior for compatibility and add a separate pure metric, for example:Semantics to define
baselinefromstartuntil the first valid point;end;end <= start;Acceptance criteria
auc()callers and behavior remain unchanged.