Skip to content

PiWindDtype/test_1 fails --check-results when run in isolation (out of ORDERED_MODELS sequence). #57

Description

@SkylordA

Issue Description

PiWindDtype is listed in ORDERED_MODELS in tests/test_model_runs.py, meant to run only after every other model in the same pytest session. Running PiWindDtype/test_1 in isolation (-k PiWindDtype) fails --check-results against expected_results/output, even on main with no code change involved.

Confirmed not a code/output issue: dispatched with -k PiWindDtype on both main and an unrelated PR branch — both failed identically, and a diagnostic sha256 hash of every generated output/* file matched exactly between the two. So the divergence is against expected_results itself, triggered by out-of-order/isolated execution, likely due to some state (JIT cache, clearcache side effect, shared session state) that expected_results implicitly depends on from earlier models running first.

Makes -k-filtered runs of PiWindDtype/test_1 unreliable for CI debugging/bisection.

Steps to Reproduce (Bugs only)

  1. pytest tests/test_model_runs.py -k PiWindDtype --check-results on a clean main checkout.
  2. Observe FAILED ...PiWindDtype/test_1 - Failed: --check-results: output differs from .../expected_results/output.
  3. Compare to a full-suite run (no -k filter, same commit) — passes.

Version / Environment information

  • OasisModels tests/test_model_runs.py (added in enhance/dtype env #46)
  • Reproduced via OasisModels CLI workflow (OasisLMF/.github/workflows/piwind-mdk.yml), workflow_dispatch with pytest_params: -k PiWindDtype
  • Python 3.10–3.14, numpy 2.5.3 / numba 0.67.0 / pandas 3.0.5
  • Works when run as part of the full ordered suite

Example data / logs

FAILED tests/test_model_runs.py::test_model_run[PiWindDtype/test_1] - Failed: --check-results: output differs from .../PiWindDtype/tests/test_1/expected_results/output

output/gul_S1_melt.csv sha256 identical on both branches tested (88755fec...0888c17), confirming outputs match each other but not expected_results.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions