With #481, the monthly MOC streamfunction in the first period after a restart is not bit for bit with the same period in a continuous run. It differs by 1–2 ulp. Everything else in the continuation is bit for bit, and the MOC is bit for bit again from the second period on.
Test. QU240, #481 at d31f221 merged with #553, on Chrysalis with Intel (oneAPI 2025.2), OpenMPI, 36 tasks. A one-year run was continued from its 0001-03-01 restart for 61 days, with StartTime set to the restart time. Then the March and April files were compared with the full run's.
| file |
March (first period after restart) |
April |
| monthly means |
BFB |
BFB |
| history snapshots |
BFB |
BFB |
GeomZInterface_HorzMean, MOCLatBinBoundaries |
BFB |
BFB |
MOC_streamfunction_Global_TimeMean1Month |
373 of 3660 values differ, 1–2 ulp (max |diff| 1.4e-14 on values up to 83 Sv) |
BFB |
@brian-oneill repeated the test on Frontier with craygnu and got bit-for-bit results for every analysis output. That suggests an Intel optimization, not missing restart state. It is still odd that the difference shows up only in the first period after a restart and only in the MOC chain.
Plan. We're tolerating this for now, because we don't expect to rely much on analysis restarts. Once #524 is in, we should check again, since restarts are still a little messy on develop.
Run directory on Chrysalis, including README.md and compare.py:
/lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/test_20260915/qu240-restart-from-0001-03-01
Full run: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/test_20260914/qu240-monthly-avgs/ocean/analysis_test/QU240/forward
Drafted by Claude Code for @xylar. Please treat the analysis as AI-authored and check it.
With #481, the monthly MOC streamfunction in the first period after a restart is not bit for bit with the same period in a continuous run. It differs by 1–2 ulp. Everything else in the continuation is bit for bit, and the MOC is bit for bit again from the second period on.
Test. QU240, #481 at d31f221 merged with #553, on Chrysalis with Intel (oneAPI 2025.2), OpenMPI, 36 tasks. A one-year run was continued from its
0001-03-01restart for 61 days, withStartTimeset to the restart time. Then the March and April files were compared with the full run's.GeomZInterface_HorzMean,MOCLatBinBoundariesMOC_streamfunction_Global_TimeMean1Month@brian-oneill repeated the test on Frontier with
craygnuand got bit-for-bit results for every analysis output. That suggests an Intel optimization, not missing restart state. It is still odd that the difference shows up only in the first period after a restart and only in the MOC chain.Plan. We're tolerating this for now, because we don't expect to rely much on analysis restarts. Once #524 is in, we should check again, since restarts are still a little messy on
develop.Run directory on Chrysalis, including
README.mdandcompare.py:Full run:
/lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/test_20260914/qu240-monthly-avgs/ocean/analysis_test/QU240/forwardDrafted by Claude Code for @xylar. Please treat the analysis as AI-authored and check it.