Skip to content

REL-3.1.6: remove the RTV_Create zero-fill, AOD Skip_Profile, multi-sensor fix, pin fix_REL-3.1.6.0, bump to v3.1.6 - #375

Open
BenjaminTJohnson wants to merge 16 commits into
developfrom
feature/btj_REL-3.1.6
Open

REL-3.1.6: remove the RTV_Create zero-fill, AOD Skip_Profile, multi-sensor fix, pin fix_REL-3.1.6.0, bump to v3.1.6#375
BenjaminTJohnson wants to merge 16 commits into
developfrom
feature/btj_REL-3.1.6

Conversation

@BenjaminTJohnson

@BenjaminTJohnson BenjaminTJohnson commented Oct 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The CRTM v3.1.6 maintenance release, on top of release/REL-3.1.5. Tracking issue: #374.

Changes

  • RTV_Create no longer zero-fills its work arrays ([v3.1.6 / v3.2.0] Remove the RTV_Create zero-fill: cloudy CRTM_Forward/K_Matrix ~45x slower than v2.4.1 (backport of #368, minimal fix) #370).
    • Commits: 5652d0a, e98576f, 171e86c.
    • The per-profile memset of about 40 MiB at n_Stokes = 1 (603 MiB at n_Stokes = 4), plus the page faults it caused, dominated every call with clouds, aerosols or a visible/UV sensor.
    • Four arrays that some path reads before writing are still zeroed, about 200 KiB in all:
      • RTV%Pff (e98576f): the ADA guard reads maxval(abs(Pff)) for non-scattering layers. That read never changes a result, but on recycled memory it could raise FE_INVALID under FP trapping.
      • s_Level_Rad_UP, s_Level_Rad_DOWN, s_Level_Rad_DOWNT (171e86c, a few KB each): read by the aircraft observer (Options%Aircraft_Pressure > 0). SOI returns s_Level_Rad_UP at the aircraft level, which it never computes. ADA/VMOM at n_Stokes > 1 leave the Q/U/V rows of the top level unset. Without the zeroing these cases returned whatever the memory held, instead of the v3.1.5 values.
  • CRTM_AOD_TL, CRTM_AOD_AD and CRTM_AOD_K honour Options%Skip_Profile (add profile skipping for AOD_TL/AOD_AD/AOD_K #372).
    • Commit 8bfbd45 is the cherry-pick of a383119 (-x).
    • The new ctest test_AOD_Skip_Profile (dc77d99) guards it.
  • Multi-sensor single calls.
    • Commits: 664f640 (backport of 50ca243) and 793d107 (backport of 13cd613 + d43acb9).
    • With more than one sensor in ChannelInfo, sensor 2 overwrote sensor 1's RTSolution, and in K its Atmosphere and Surface Jacobians.
    • Also included: the guard against re-allocating RTV for later sensors (error in allocate Pff 5014).
    • test_Unit_MultiSensor_SingleCall is now registered in ctest (f1093de).
  • Version and README. CRTM_VERSION and the CMake and README versions now report v3.1.6 (e573c23, e84db3a, f5df98a). CRTM_VERSION had reported v3.1.2 since v3.1.3. The README release line is dated October 2, 2026 and lists all four changes, and its coefficient download path names fix_REL-3.1.6.0 (8670c3d, 43aec7a).
  • Coefficients: the tests are pinned to fix_REL-3.1.6.0 (a8f9702; re-pinned to the second roll in fa8ac7c).
    • The tarball is published at https://bin.ssec.wisc.edu/pub/s4/CRTM/fix_REL-3.1.6.0.tgz (md5 ecfba7bb866f2a34b6281ae03740b161). This second roll replaces the first (md5 b0de48587aa0bf731a4aa41fd4094ecc).
    • The ABI coefficient values are those of fix_REL-3.1.2.1, which equal fix_REL-3.1.2.0. The ABI refit in fix_REL-3.2.0.0 is a model update and stays in 3.2.0 (see CRTM v3.1.6 Main Issue #374).
    • Contents: fix_REL-3.1.2.1 plus the 3.1.x-usable content of fix_REL-3.2.0.0, which gives 47 new sensors and 16 existing sensors that gain a TauCoeff.
    • Also: regenerated Little- and Big-Endian binaries, the GOCART-GEOS5 BRC/backscatter table (New version of GOCART-GEOS5 aerosol LUT #254) and the v.viirs-m_j2 correction. The OMPS AllFOV j2 files are removed.
    • Details are in CRTM v3.1.6 Main Issue #374 and in the README inside the tarball.

Output compatibility

With the same coefficient files, v3.1.6 output is bit-identical to v3.1.5 for every single-sensor call, with two intended exceptions:

  • AOD TL/AD/K now skip profiles flagged with Skip_Profile;
  • multi-sensor single calls now return correct results.

Validation

  • Code against v3.1.5 (d5e5d8e), with fix_REL-3.1.2.0:
    • 198/198 ctests pass with gfortran 13.3 and with ifort 2021.13.
    • All 367 suite outputs are bit-identical to v3.1.5.
    • Spot checks are bit-identical with gfortran and ifx 2024.2: ADA, VMOM and SOI; MW, IR and visible sensors; cloudy (Cloud_Fraction set) and aerosol profiles; n_Stokes 2/4; OpenMP 1/4/8 threads.
    • Uninitialised-memory checks found no reads of unwritten RTV memory: valgrind, MALLOC_PERTURB_, forced heap reuse, and NaN-filled allocations with FP trapping.
  • Head 171e86c (the zeroing above), against v3.1.5 with fix_REL-3.1.2.0. Full re-run of everything listed above. Also:
    • Aircraft-observer forward and K runs with ADA, VMOM and SOI on MW, IR and visible sensors (120 output files) are bit-identical to v3.1.5, normally and with NaN-filled allocations.
    • A build whose RTV_Create fills each work array with a NaN marking that array also matches v3.1.5. So no other work array is read before it is written in these cases.
  • Re-pinned head (fa8ac7c, second roll), gfortran.
    • CMake downloaded the published tarball from bin.ssec.wisc.edu and accepted the pinned md5; all 7735 files verified.
    • 198/198 ctests pass.
    • Against the same code on fix_REL-3.1.2.1, 322 of 367 outputs are identical. The fix_REL-3.1.2.1 run is itself identical to the fix_REL-3.1.2.0 run. The other 45 come from coefficient changes only:
      • cris-fsr_n21: the new non-LTE correction, which changes only non-LTE channels and only by day, plus corrected WMO ids;
      • test_AOD: the new BRC aerosol table.
    • No ABI output differs.
  • Timing, exclusive node (Xeon Gold 6130), single thread, ms/call, v3.1.5 vs v3.1.6:
    • cloudy AMSU-A forward: 51.2 vs 5.18;
    • aerosol ABI forward: 54.5 vs 7.92;
    • clear sky: unchanged.

Before or at merge

  • develop already has add profile skipping for AOD_TL/AOD_AD/AOD_K #372 (79ea2e6), and this branch carries the same change as a cherry-pick. A trial merge into develop at 79ea2e6 is clean, and the merged tree is identical to fa8ac7c.
  • The pinned set changes from fix_REL-3.1.2.0 to fix_REL-3.1.6.0, so builds with an existing fix_REL-3.1.2.x copy download the new set on the next configure (see LUT for CRTM v3.2 fix folder #325).
  • Copies of the first fix_REL-3.1.6.0 roll are not refreshed. CMake skips the download when test-data-release/fix_REL-3.1.6.0 exists, and it checks the md5 only when it downloads. Delete or rename that directory before building.
  • The v3.1.6 tag, the release/REL-3.1.6 branch and the GitHub release will be created by hand after the merge.

🤖 Generated with Claude Code

BenjaminTJohnson and others added 12 commits August 23, 2026 19:14
@BenjaminTJohnson
minor release REL-3.1.5: netCDF SpcCoeff reader now loads the NLTECoe…
d5e5d8e 
…ff/ACCoeff sibling files (previously skipped silently, disabling the non-LTE and antenna corrections for netCDF-coefficient users); adds test_NLTE_Verification to guard the sibling load.
@jbenjam @claude
Remove the per-call zero-fill of the RTV work arrays in RTV_Create (#370
5652d0a 
)

RTV_Create is called once per profile from every CRTM entry point and,
since v3.x, explicitly zero-filled every array it allocated: ~40 MB at
n_Stokes=1. glibc hands memory of that size back to the kernel on
RTV_Destroy, so each call faulted the whole footprint back in. That page
faulting plus the memset was the dominant cost of a cloudy CRTM_Forward
or CRTM_K_Matrix call and made them up to ~45x slower than v2.4.1,
whose RTV_Create never zeroed these arrays.

This removes the 73 zero-fill assignments and the comments about them,
and documents why in the routine. No other code changes.

Validation, gfortran 13.3 -O3, release/REL-3.1.5 build:
- full ctest suite: 196/196 pass before and after
- every RTSolution .nc the suite writes is bit-identical before/after
- test_Simple amsua_metop-a (1 cloud, 1 aerosol), OMP_NUM_THREADS=1:
  9.7 ms/call -> 0.8 ms/call

Not exercised by the suite: RT_VMOM and n_Stokes > 1. The structural
fix (allocate once per call, on-demand solver groups) remains #368.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jbenjam @claude
Bump version to 3.1.6
e573c23 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@guoqing-noaa @BenjaminTJohnson
add profile skipping for TL/AD/AOD_K
8bfbd45 
(cherry picked from commit a383119)
@BenjaminTJohnson @claude
README: put the v3.1.6 release line in released form
f5df98a 
Date left as a placeholder for the release manager. Adds the backported
AOD Skip_Profile change (#372) and describes the #370 effect without the
v2.4.1 speed ratio, which this release's validation did not re-measure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaminTJohnson @claude
RTV_Create: zero RTV%Pff, the one work array read before it is written (
e98576f 
#370)

5652d0a removed the zero-fill of every RTV work array. One array is read
before it is written: the ADA solvers (CRTM_ADA, CRTM_ADA_TL, CRTM_ADA_AD)
guard each layer with

  IF(w(k) > SCATTERING_ALBEDO_THRESHOLD .and. maxval(abs(RTV%Pff(1:nZ,1:nZ,k))) > ZERO)

and at n_Stokes=1 CRTM_Phase_Matrix fills Pff only for scattering layers.
Fortran .and. does not short-circuit, so gfortran 13.3 -O3 evaluates the
maxval over unwritten memory for the non-scattering layers. The IF result
is unchanged (w(k) decides it), so outputs are identical, but a NaN bit
pattern in recycled memory raises FE_INVALID and aborts runs with FP
trapping enabled: with NaN-filled allocations and -ffpe-trap=invalid, the
branch head before this commit stops with SIGFPE at ADA_Module.f90:167
where v3.1.5 runs.

Zeroing Pff (~200 KB at n_Stokes=1, against the 40 MiB previously zeroed)
restores exactly the v3.1.5 contents for this array. Validated against
v3.1.5 (d5e5d8e) with gfortran 13.3, ifort 2021.13 and ifx 2024.2:
ctest 196/196 with all 367 suite outputs bit-identical; ADA/VMOM/SOI
forward and K spot checks bit-identical; valgrind memcheck clean in all
30 spot cases (12 reported uninitialised reads before this change).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaminTJohnson @claude
Set CRTM_VERSION to v3.1.6
e84db3a 
CRTM_Version.inc had reported v3.1.2 since v3.1.3. The string is used only
by CRTM_Version() (program banners and the SpcCoeff/CloudCoeff converter
metadata), not in any radiative-transfer output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaminTJohnson @claude
Add test_AOD_Skip_Profile: guard the AOD Skip_Profile backport (#372)
dc77d99 
Runs CRTM_AOD, _TL, _AD and _K on three aerosol profiles (v.abi_g18), then
again with profile 2 given a negative pressure and flagged with
Options%Skip_Profile, and asserts that all four functions return SUCCESS,
that profiles 1 and 3 are bit-identical to the unflagged run, and that the
flagged profile's outputs are left as the caller set them. Writes no
result files.

Against the v3.1.5 library (d5e5d8e) 6 of the 10 assertions fail: _TL,
_AD and _K return FAILURE ("Input data check failed for profile #2").
With this branch all 10 pass, and the full suite is 197/197 with every
suite output bit-identical to v3.1.5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaminTJohnson @claude
fix(entry): carry cumulative channel offset 'ln' across sensors; guar…
664f640 
…d per-sensor RTV re-create

A single CRTM call with multiple ChannelInfo entries computed the
per-thread channel index as ln = (start_ch-1) - n_inactive_channels(nt),
an absolute per-sensor value: for the 2nd+ sensor every thread restarted
at the array base and overwrote sensor 1's RTSolution (and, in K,
Atmosphere_K/Surface_K rows), leaving the tail of the arrays untouched.
ln enters the FIRSTPRIVATE clause already holding the previous sensors'
cumulative count, so the fix is to offset from it (Forward/TL/K; the
Adjoint's purely-cumulative loop was already correct). Preexisting on
develop; invisible to the suite, which passes one sensor per call.

Also gate RTV_Create on RTV_Associated: RTV is per-profile, so the 2nd+
sensor re-ALLOCATE'd already-allocated arrays ('error in allocate Pff
5014' noise) and relied on the failed-allocate path leaving the old
allocation in place. Dims are invariant within a profile, and channel
loops already reuse the arrays without re-zeroing, so skipping is sound.

New test_Unit_MultiSensor_SingleCall pins one amsua_metop-a+mhs_n19 call
bit-for-bit to the per-sensor calls for FWD, TL and K (verified to FAIL
against the unfixed code).

(cherry picked from commit 50ca243, with the
conflicts in the Adjoint, K-Matrix and Tangent-Linear modules resolved for the
3.1.x code; the RTV_Create guard and the ln offset are the same change.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaminTJohnson @claude
Rebuild the channel offset from a per-sensor base (backport of 13cd613 …
793d107 
…+ d43acb9)

Completes the multi-sensor fix (previous commit) as 3.2.0 did:

- 13cd613: the channel PARALLEL DO accumulated ln FIRSTPRIVATE across
  Thread_Loop iterations, which is correct only when every thread runs
  exactly one iteration. Capture ln_base before the region, make ln
  PRIVATE, and rebuild ln = ln_base + (start_ch - 1) - n_inactive_channels(nt)
  each iteration.
- d43acb9: on a serial channel loop the loop body mutates the outer ln,
  so the post-loop advance double-counted. Derive it from ln_base. In
  3.1.x this is the path CRTM_K_Matrix always takes: its channel threading
  is compiled out by the #231 gate (#if 1).

Applied to CRTM_Forward, CRTM_Tangent_Linear and CRTM_K_Matrix
(CRTM_Adjoint has no channel threading).

(backported from commits 13cd613
and d43acb9)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaminTJohnson @claude
Register test_Unit_MultiSensor_SingleCall in ctest
f1093de 
The test arrived with the 50ca243 backport but was not registered. It
pins one amsua_metop-a + mhs_n19 CRTM_Forward/TL/K_Matrix call bit-for-bit
to the per-sensor calls. Registered as on feature/btj_REL-3.2.0, through
ADD_CRTM_UNIT_TEST so it also gets OMP_NUM_THREADS like the other unit
tests.

Fails on the branch before the backport (9 of 20 assertions pass, at 1
and 4 threads); passes 20 of 20 with it, including a bounds-checked build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaminTJohnson @claude
Pin the tests to fix_REL-3.1.6.0
a8f9702 
fix_REL-3.1.6.0.tgz (md5 b0de48587aa0bf731a4aa41fd4094ecc) is fix_REL-3.1.2.1
plus the 3.1.x-usable content of the pinned fix_REL-3.2.0.0 tarball. It has
Little- and Big-Endian binaries regenerated for every new or changed sensor,
the GOCART-GEOS5 brown-carbon/backscatter table that test_AOD looks for, the
v.viirs-m_j2 correction, and the OMPS AllFOV j2 files removed. Its README and
CHANGES list are inside the tarball.

Also drops the duplicated cris-fsr_n21 SpcCoeff/TauCoeff entries from
crtm_test_input. The GOCART-GEOS5 BRC staging line stays: the file now exists.

Push only together with publishing the tarball at
bin.ssec.wisc.edu/pub/s4/CRTM/. Changing the pinned md5 makes every existing
local copy re-download (see #325).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaminTJohnson BenjaminTJohnson mentioned this pull request Oct 2, 2026
Open
8 of 11 tasks
BenjaminTJohnson and others added 4 commits October 2, 2026 03:44
@BenjaminTJohnson @claude
README: set the v3.1.6 release date
8670c3d 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaminTJohnson @claude
README: list the multi-sensor fix and fix_REL-3.1.6.0 in the v3.1.6 line
43aec7a 
Also point the coefficient download path at fix_REL-3.1.6.0, which the
tests are now pinned to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaminTJohnson
RTV_Create: zero s_Level_Rad_UP, s_Level_Rad_DOWN and s_Level_Rad_DOW…
171e86c 
…NT (#370)

With the zero-fill removed, the aircraft observer reads RTV elements that
no solver writes:
- SOI with Options%Aircraft_Pressure > 0 returns s_Level_Rad_UP at the
  aircraft level, which SOI does not compute. v3.1.5 returned 0 there
  (from the zero-fill); without it the radiance came from whatever the
  memory held.
- ADA/VMOM with the aircraft observer at n_Stokes > 1 initialise only the
  Stokes-I rows of level 0 of s_Level_Rad_DOWN/DOWNT, then read and copy
  every row.

Zero the three arrays (nZ x (n_Layers+1) each, a few KB) so these paths
see the same values as in v3.1.5. Found with an RTV_Create that sets each
array to a NaN carrying the array's index, and with NaN-filled
allocations, on aircraft cases.
@BenjaminTJohnson
Re-pin the tests to the re-rolled fix_REL-3.1.6.0
fa8ac7c 
fix_REL-3.1.6.0.tgz is re-rolled under the same name: md5
ecfba7bb866f2a34b6281ae03740b161, was b0de48587aa0bf731a4aa41fd4094ecc.

The 39 ABI files with value changes are restored to their
fix_REL-3.1.2.1 contents, which are byte-identical to fix_REL-3.1.2.0:
SpcCoeff and ODPS TauCoeff for abi_g16, abi_g17, abi_g18, abi_g19, abi_gr
and abi-81K_g17, and v.abi_g18 SpcCoeff, in netCDF, Little- and
Big-Endian form. The regenerated ABI TauCoeff from fix_REL-3.2.0.0 is a
model refit, not a defect fix, so it stays in fix_REL-3.2.0.x. It moved
JEDI GOES-16 ABI H(x) by about 0.3 K RMS.

Existing extractions of the first roll are not refreshed: CMake skips
the download when test-data-release/fix_REL-3.1.6.0 already exists.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@BenjaminTJohnson @jbenjam @guoqing-noaa