test(sr-tests): add golden-output regression test for predict() - #383
Open
dhritimandas wants to merge 1 commit into
Open
test(sr-tests): add golden-output regression test for predict()#383dhritimandas wants to merge 1 commit into
dhritimandas wants to merge 1 commit into
Conversation
Nothing pins predict()'s actual output values -- existing tests assert shapes, dtypes, and finiteness, never numeric content. A refactor to padding, block splitting, reassembly, or the argmax/threshold logic could silently change what users get. - nobrainer/sr-tests/golden/generate_golden.py: builds a tiny brain-extraction unet (channels=(4,8), strides=(2,)), briefly overfits a synthetic sphere volume, runs predict(), and writes the committed fixture. Not test_-prefixed, so pytest never collects it; run it deliberately to regenerate. - nobrainer/sr-tests/golden/golden_brain_extraction.npz: the fixture (26.4 KB) -- packed label mask + sha256, a probability probe at 4096 stride-sampled voxels, the model weights, and a meta block recording the generating environment (torch/monai/numpy/platform versions) for diagnosability. - nobrainer/sr-tests/test_golden_outputs.py: sha256 fast path (the default, non-strided prediction path reassembles via pure block assignment with no float accumulation, so the label mask is genuinely bit-stable on a fixed platform), falling back to a voxel-disagreement budget + Dice check, then a probability-probe assert_close. Every tolerance is derived from this fixture's actual measurements (object size, worst-case Dice at the budget ceiling) and commented inline, never a bare default. device="cpu" is pinned explicitly rather than left to get_device(), which would return CUDA/MPS depending on runner. monai.utils.set_determinism(seed=42) is called before every model construction and predict() call in both scripts. - nobrainer/sr-tests/golden/README.md: what the fixture contains, why each design choice, and the deliberate regeneration workflow (dry-run diff by default, --yes to write, refuses a degenerate mask). No @pytest.mark.gpu, so `-m "not gpu"` includes these tests -- they run on CPU CI with no network and no GPU.
Open
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
What: We add a golden-output regression test for
predict(). A committed fixture pins its numeric output.Why: No test checked the values that
predict()returns. Shape and dtype tests miss a changed pad, block order, or argmax. A refactor could change results silently.Fixture (
nobrainer/sr-tests/golden/)channels=(4,8),strides=(2,)) briefly overfit on an analytic sphere; weights + packed label mask + 4096-voxel probability probe +metablock (versions/platform/seed) in one 26.4 KB.npzmetavianp.mgrid— no RNG, no stored volume(40,48,56)deliberately non-multiple of the 16³ block on 2 axes, so_pad_to_multipleis always exercised (asserted)Test (
test_golden_outputs.py)torch.testing.assert_close(rtol=1e-4, atol=1e-5), every tolerance justified inlinedevice="cpu"pinned explicitly;set_determinism(seed=42)before every runbatch_sizeinvariance pinnedgpumarker — runs on CPU CI, no network, no downloadsRegeneration (
generate_golden.py)--yesto write; refuses a degenerate maskgolden/README.mdTest plan
test_croissantfailure, fixed by feat(data): DataSpec validation, Croissant fix, MPS guard #376)