Another slowdown: 5x slower over 6 years

[mark-petersen]

I ran SOMA 32km using Sept 21 develop 0c3353c with the following items merged in:

• KPP add KPP mixing #512
• Monthly averages Add monthly averages #553
• New DVD code by Cage Mitchell
• Note develop already includes the previous slowdown fix Fix memory growth during configuration retrieval #541 merged Sept 10.

Simulations ran for 10 years with a restart and resubmission at year 6. Here is the time between restart files recorded by the unix time stamp:

This is on perlmutter cpu with gnu. I am running on a single node using 128 cores. run directory at:

/pscratch/sd/m/mpeterse/runs/260925-soma-long-runs/32km

I observed this behavior on all four resolutions.

Slowdown could be on develop or from any of the PRs above. Testing that now.

[mark-petersen]

I ran today's develop 39689a9 using the SOMA test case at 32km with all IO off except monthly restart files. On perlmutter-cpu gnu with -DOMEGA_BUILD_TYPE=Release

/pscratch/sd/m/mpeterse/runs/260928-test-kpp-soma/develop-cpu-1

It ran 6 years in 2.5WC hours. This had a 4x slowdown:

I restarted at year 6 and it is back down again, so this definitely accumulates during the run.

[mark-petersen]

I compared the timer output between a 6 year run and 3 month run. The proportional time increase is in the Omega:Stepper:doStep. That tells us it is probably a memory leak in the PDE solver, not in IO.

[mark-petersen]

After some experimentation, I found that if I only turn off these terms from the previous run on develop

ThicknessFluxTendencyEnable: false
TracerHorzAdvTendencyEnable: false
VelocityVertAdvTendencyEnable: false

that the speed-up disappears.

If I try to toggle these individually rather than as a group, the simulation hits a nan within a day.

[mwarusz]

@mark-petersen

You could try #584 A timer that gets opened on every step, but never closed, might explain this gradual slowdown.

[mark-petersen]

Thanks @brian-oneill and @mwarusz! #584 appears to have fixed the problems! I ran the identical above using develop and SOMA with no IO but with #584 merged in. There is now no slowdown on this 5 year run: