Skip to content

Add state-aware predictive scores - #90

Draft
jonathanberthias wants to merge 3 commits into
migration/sampleable-capabilityfrom
migration/explicit-state-predictive-scores
Draft

jonathanberthias wants to merge 3 commits into
migration/sampleable-capabilityfrom
migration/explicit-state-predictive-scores

Conversation

@jonathanberthias

@jonathanberthias jonathanberthias commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

The legacy CRPS, Brier, and quantile scores read distribution values implicitly and cannot evaluate an explicit parameter state. The legacy CRPS also squares the difference from the empirical CDF, which is not the mean score of individual forecasts when observations have different predictions.

This PR adds state-aware CRPS, Brier, and quantile scores for univariate continuous models. Each takes a structural model, scalar or one-dimensional observations, and an optional explicit state, so the same function can score a candidate during optimization or inspect a fitted state. Forecasts must be scalar or aligned with observations. CRPS averages individual scores and splits quadrature at observations and finite support endpoints and medians, so narrow bounded forecasts are resolved.

AIC and BIC remain separate post-fit diagnostics because their parameter penalties are constant for a fixed model. Discrete threshold events need their own probability capability before Brier scoring is generalized beyond continuous laws.

@jonathanberthias
jonathanberthias force-pushed the migration/explicit-state-predictive-scores branch from 33e8bc7 to 9527951 Compare September 27, 2026 20:27
@jonathanberthias
jonathanberthias force-pushed the migration/sampleable-capability branch from 3fda28a to 48923fa Compare September 27, 2026 20:27
@jonathanberthias
jonathanberthias force-pushed the migration/explicit-state-predictive-scores branch from dd394e5 to bd81cb8 Compare October 2, 2026 21:05
@jonathanberthias
jonathanberthias force-pushed the migration/sampleable-capability branch 2 times, most recently from 5490832 to a4e6af2 Compare October 2, 2026 22:25
@jonathanberthias
jonathanberthias force-pushed the migration/explicit-state-predictive-scores branch from bd81cb8 to 22b2063 Compare October 2, 2026 22:25
@jonathanberthias
jonathanberthias force-pushed the migration/sampleable-capability branch from a4e6af2 to 8def56e Compare October 3, 2026 11:02
@jonathanberthias
jonathanberthias force-pushed the migration/explicit-state-predictive-scores branch from 22b2063 to 2abc681 Compare October 3, 2026 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant