Add state-aware predictive scores - #90
Draft
jonathanberthias wants to merge 3 commits into
Draft
jonathanberthias wants to merge 3 commits into
jonathanberthias wants to merge 3 commits into
Conversation
jonathanberthias
force-pushed
the
migration/explicit-state-predictive-scores
branch
from
September 27, 2026 20:27
33e8bc7 to
9527951
Compare
jonathanberthias
force-pushed
the
migration/sampleable-capability
branch
from
September 27, 2026 20:27
3fda28a to
48923fa
Compare
jonathanberthias
force-pushed
the
migration/explicit-state-predictive-scores
branch
from
October 2, 2026 21:05
dd394e5 to
bd81cb8
Compare
jonathanberthias
force-pushed
the
migration/sampleable-capability
branch
2 times, most recently
from
October 2, 2026 22:25
5490832 to
a4e6af2
Compare
jonathanberthias
force-pushed
the
migration/explicit-state-predictive-scores
branch
from
October 2, 2026 22:25
bd81cb8 to
22b2063
Compare
jonathanberthias
force-pushed
the
migration/sampleable-capability
branch
from
October 3, 2026 11:02
a4e6af2 to
8def56e
Compare
jonathanberthias
force-pushed
the
migration/explicit-state-predictive-scores
branch
from
October 3, 2026 11:03
22b2063 to
2abc681
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The legacy CRPS, Brier, and quantile scores read distribution values implicitly and cannot evaluate an explicit parameter state. The legacy CRPS also squares the difference from the empirical CDF, which is not the mean score of individual forecasts when observations have different predictions.
This PR adds state-aware CRPS, Brier, and quantile scores for univariate continuous models. Each takes a structural model, scalar or one-dimensional observations, and an optional explicit state, so the same function can score a candidate during optimization or inspect a fitted state. Forecasts must be scalar or aligned with observations. CRPS averages individual scores and splits quadrature at observations and finite support endpoints and medians, so narrow bounded forecasts are resolved.
AIC and BIC remain separate post-fit diagnostics because their parameter penalties are constant for a fixed model. Discrete threshold events need their own probability capability before Brier scoring is generalized beyond continuous laws.