Skip to content

Bug: Score of multiple models fails if observations are not in each comparer - #526

Merged
ecomodeller merged 13 commits into
mainfrom
fix/score_multiple_models
Oct 6, 2025
Merged

ecomodeller merged 13 commits into
mainfrom
fix/score_multiple_models

Conversation

@ecomodeller

@ecomodeller ecomodeller commented Sep 23, 2025 •

Copy link
Copy Markdown
Member
import modelskill as ms
import pandas as pd

time = pd.date_range("2000", periods=2)
cmp1 = ms.match(
    obs=ms.PointObservation(pd.Series([0, 0], index=time), name="foo"),
    mod=ms.PointModelResult(pd.Series([0, 0], index=time), name="bar"),
)
cmp2 = ms.match(
    obs=ms.PointObservation(pd.Series([0, 0], index=time), name="baz"),
    mod=ms.PointModelResult(pd.Series([0, 0], index=time), name="qux"),
)
# TODO should it be allowed to combine two comparers with different models, maybe not?
cc = ms.ComparerCollection([cmp1, cmp2])
cc.skill(metrics="rmse")
image

So far so good, but if I call

cc.score() 😟

it fails with a cryptical message since the weights vector is of the wrong length, if each comparers doesn't contain each observation.

        if len(data) != len(index):
>           raise ValueError(
                "Length of values "
                f"({len(data)}) "
                "does not match length of index "
                f"({len(index)})"
            )
E           ValueError: Length of values (4) does not match length of index (2)

This PR fixes this problem, and does some more refactoring related to weighted scoring.

@ecomodeller
ecomodeller marked this pull request as ready for review September 23, 2025 20:55
@ecomodeller ecomodeller changed the title Fix score of multiple models Bug: Score of multiple models fails if observations are not in each comparer Sep 24, 2025

@ryan-kipawa ryan-kipawa left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good :)

I assume "should it be allowed to combine two comparers with different models, maybe not?" is just a general comment outside the scope of the PR.

@ecomodeller

Copy link
Copy Markdown
Member Author

Looks good :)

I assume "should it be allowed to combine two comparers with different models, maybe not?" is just a general comment outside the scope of the PR.

Indeed, thinking out "loud"...

@ecomodeller
ecomodeller merged commit 7ea4e4f into main Oct 6, 2025
3 checks passed
@ecomodeller
ecomodeller deleted the fix/score_multiple_models branch October 6, 2025 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants