Fix timer failures from unmatched Pacer timer

[brian-oneill]

A Pacer::start without a matching Pacer::stop in Tendencies::computeTracerTendenciesOnly is causing cascading timer failures in the Omega run loop (#583). This PR fixes the bug by removing the duplicate Pacer::start("Tend::tracerHorzAdv") and moving the corresponding Pacer::stop outside the FCT/non-FCT branching, so the resulting timer wraps around the entire tracer horizontal advection block and is properly implemented for both TracerHorzAdv.FCT = true and TracerHorzAdv.FCT = false.

Checklist

• Linting

  • Pre-commit run locally

• Building

  • CMake build does not produce any new warnings from changes in this PR

• Testing

  aurora, oneapi-ifx, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  chrysalis, oneapi-ifx, openmpi

  • CTests Pass
  • Polaris omega_pr Pass

  frontier, craygnu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  frontier, craygnu-mphipcc, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  pm-cpu, gnu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  pm-gpu, gnugpu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

• Provide relevant details in a comment to the PR titled Testing with the following:

  • Which machines CTest unit tests
    have been run on and indicate that are all passing.
  • The Polaris omega_pr test suite
    has passed, using the Polaris e3sm_submodules/Omega baseline
  • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
  • Indicate "All tests passed" or document failing tests
  • Document testing used to verify the changes including any tests that are added/modified/impacted.

Fixes #583
Fixes #580

[mark-petersen]

This PR definitely fixes a slow-down problem I had.

Before
I ran today's develop 39689a9 using the SOMA test case at 32km with all IO off except monthly restart files. On perlmutter-cpu gnu with -DOMEGA_BUILD_TYPE=Release

After
Same as above but with this PR merged in

[mark-petersen]

Testing

CTest unit tests Perlmutter CPU

• Machine: perlmutter-cpu
• Compiler: gnu
• Build type: Release
• Result: All tests passed
• Logs: /pscratch/sd/m/mpeterse/omega-builds/od-cpu-260928/build/test/logs

CTest unit tests Perlmutter GPU

• Machine: perlmutter-gpu
• Compiler: gnugpu
• Build type: Release
• Result: All tests passed
• Logs: /pscratch/sd/m/mpeterse/omega-builds/od-gpu-260928/build/test/logs

CTest unit tests Frontier CPU

• Machine: frontier-cpu
• Compiler: craygnu
• Build type: Release
• Result: All tests passed
• Logs: /lustre/orion/cli115/scratch/mpetersen/omega-builds/test_omega_cpu/build/test/logs

CTest unit tests Frontier GPU

• Machine: frontier-gpu
• Compiler: craygnu-mphipcc
• Build type: Release
• Result: All tests passed
• Logs: /lustre/orion/cli115/scratch/mpetersen/omega-builds/test_omega_gpu/build/test/logs

[mark-petersen]

Tested Polaris omega_pr on

• Machine: perlmutter-cpu
• Compiler: gnu
• Build type: Release
• All tests pass

[mark-petersen]

Thanks @brian-oneill for the quick bug fix!