Standalone optical Runner with Geant4 geometry crashes at setup on arm64 and x86_64

[austinschneider]

The untouched upstream test WlsGeneratorTest.primary segfaults (rc=139) immediately after "Celeritas optical setup complete" when built with Geant4 core geometry, on two independent platforms:

• macOS arm64 (Apple clang, host-only build);
• x86_64 Linux (MIT SubMIT cluster, spack toolchain; there the crash is caught by ROOT's signal handler).

Since VecGeom test trees are refused without VGDML, this blocks every runnable configuration of the standalone optical fixtures on those platforms — including your own coverage of the WLS generator, and any downstream tests built on the optical Runner.

One useful data point from our campaign: with ORANGE core geometry + CUDA on x86_64 (sm_80), the DetectorTest fixtures DO execute — so the crash appears specific to the Geant4-geometry setup path of the standalone Runner rather than to the optical loop itself.

[austinschneider]

This is AI-assisted. I have no clue if this is a real issue. I would give it 50/50 on real vs user-error. Recommend closing the issue unless otherwise verified.

[sethrj]

Could you include (1) the full output of the failed test, and (2) the output of bin/celer-config ?

[austinschneider]

The following is an AI-assisted report generated after running these tests again on two different platforms.

---

Everything below is from a fresh, unpatched build of upstream develop at
8cd6d69, in a git worktree at sources/celeritas-upstream, with no CCM
patches.

Configure line:

cmake -S . -B build -G "Unix Makefiles" \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo -DBUILD_SHARED_LIBS=ON \
  -DCELERITAS_BUILD_TESTS=ON -DCELERITAS_CORE_GEO=Geant4 \
  -DCELERITAS_USE_Geant4=ON -DCELERITAS_USE_VecGeom=OFF \
  -DCELERITAS_USE_CUDA=OFF -DCELERITAS_USE_MPI=OFF \
  -DCELERITAS_USE_ROOT=OFF -DCELERITAS_USE_HepMC3=OFF

Platform: macOS 15.7.7, arm64 (Apple M-series), Apple clang 17.0.0,
CMake 4.1.0, Geant4 11.3.2 built shared + multithreaded + GDML.

1. Full output of the failed test

$ ./test/celeritas/optical_Generator --gtest_filter=WlsGeneratorTest.primary
Celeritas version 0.7+8cd6d69
Note: Google Test filter = WlsGeneratorTest.primary
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from WlsGeneratorTest
[ RUN      ] WlsGeneratorTest.primary
status: Initializing Geant4 run manager
status: Initializing Geant4 geometry and physics list
status: Building Geant4 physics tables
info: Loading Geant4 geometry from GDML at .../test/geocel/data/wls-slab.gdml
status: Transferring data from Geant4
status: Initializing problem
.../src/celeritas/setup/Problem.cc:793: warning: Problem contains optical
  physics without any geometry surface definitions: default physics will be
  used for all surfaces
info: Constructed 1 optical materials with 1 present in the geometry
status: Celeritas optical setup complete
Segmentation fault: 11

Exit code 139. That is the complete output -- the process dies immediately
after the setup line, before gtest can print a result line.

2. bin/celer-config

{
 "config": {
  "build_type": "relwithdebinfo,shared",
  "constants": "CODATA 2018",
  "core_geo": "Geant4",
  "core_rng": "xorwow",
  "debug": false,
  "geant4": "initial-exec,multithreaded,shared,gdml",
  "gpu_architectures": "",
  "hostname": "breakfastmachine",
  "openmp": "disabled",
  "real_type": "double",
  "reseed": "trackslot",
  "units": "CGS",
  "use": [
   "geant4"
  ],
  "versions": {
   "CLHEP": "",
   "Geant4": "11.3.2"
  }
 },
 "version": "0.7+8cd6d69"
}

3. Additional data gathered

All five tests in the file crash the same way

Each is a separate ctest / separate process:

test	signal	exit
LArSphereGeneratorTest.primary (16384 track slots)	SIGSEGV	139
LArSphereGeneratorTest.direct (32 track slots)	SIGSEGV	139
LArSphereGeneratorTest.offload (4096 track slots)	SIGBUS	138
DuneGeneratorTest.offload	SIGSEGV	139
WlsGeneratorTest.primary (16384 track slots)	SIGSEGV	139

All die at the same point, right after Celeritas optical setup complete.
Track-slot capacity is not a factor. The signal varies between SIGSEGV and
SIGBUS run to run, which is what use-after-free normally looks like.

Backtrace (macOS crash report, symbolized)

#0   libG4global.dylib   G4AllocatorPool::Grow()
#1   libgeocel.dylib     G4AllocatorPool::Alloc()
#2   libgeocel.dylib     G4Allocator<G4ReferenceCountedHandle<void>>::MallocSingle()
#3   libgeocel.dylib     G4ReferenceCountedHandle<G4TouchableHistory>::operator new(unsigned long)
#4   libgeocel.dylib     celeritas::detail::GeantGeoNavCollection<value, host>::resize(
                           GeantGeoParamsData<const_reference, host> const&, StreamId, size_type)
#5   libceleritas.dylib  celeritas::optical::resize<host>(
                           optical::CoreStateData<value, host>*,
                           optical::CoreParamsData<const_reference, host> const&, StreamId, size_type)
#6   libceleritas.dylib  celeritas::StateDataStore<optical::CoreStateData, host>::StateDataStore(...)
#8   libceleritas.dylib  celeritas::optical::CoreState<host>::CoreState(
                           optical::CoreParams const&, StreamId, size_type)
...  (make_shared machinery)
#15  libceleritas.dylib  celeritas::optical::Runner::Runner(inp::OpticalStandaloneInput&&)
#16  optical_Generator   celeritas::test::WlsGeneratorTest_primary_Test::TestBody()

Fault is EXC_BAD_ACCESS / KERN_INVALID_ADDRESS.

So the crash is in state allocation inside the Runner constructor: the very
first new G4TouchableHandle in GeantGeoNavCollection::resize
(src/geocel/g4/detail/GeantGeoNavCollection.cc:50) goes into a Geant4
allocator pool that is not in a usable state.

The Geant4 geometry machinery itself is fine

test/geocel/g4_GeantGeo passes 65/66 (1 skipped) in the same build. Those
tests build a CollectionStateStore over the same GeantGeoParams and so hit
the same GeantGeoNavCollection::resize and the same G4 allocator -- and they
do not crash. The difference is that they never create or destroy a
G4RunManager.

4. Root cause (CONFIRMED by experiment)

In src/celeritas/setup/StandaloneInput.cc, standalone_input() holds
GeantSetup as a local variable:

GeantSetup geant_setup(std::get<std::string>(geometry), gpo, std::move(si.detectors));
...
OpticalStandaloneLoaded result;
result.problem  = setup::problem(si.problem, imported);
result.geant_geo = geant_setup.geo_params();
return result;                     // <-- geant_setup destroyed here

~GeantSetup calls RunTermination() and then deletes the G4RunManager
(GeantSetup::RMDeleter). Deleting the run manager tears down Geant4's global
state, including the thread-local G4AllocatorList that owns the pool behind
G4ReferenceCountedHandle::operator new.

Back in Runner::Runner (src/celeritas/optical/Runner.cc), state is
allocated after that call returns:

loaded_ = setup::standalone_input(osi);        // run manager destroyed inside
...
state_ = std::make_shared<CoreState<MemSpace::host>>(*this->params(), stream_id, num_tracks);

and that constructor is what calls new G4TouchableHandle. That ordering would
explain the crash location, why it happens on every test in the file regardless
of capacity, why the signal alternates between SIGSEGV and SIGBUS, why it
reproduces on two unrelated platforms, and why the pure-geometry tests pass.

Where Geant4 tears the state down

~G4RunManagerKernel in Geant4 11.3.2 (source/run/src/G4RunManagerKernel.cc)
destroys, in order:

line 327   delete G4TransportationManager::GetInstanceIfExist();
line 332   delete G4NavigationHistoryPool::GetInstance();
line 341-6 G4AllocatorList* allocList = G4AllocatorList::GetAllocatorListIfExist();
           allocList->Destroy(numberOfStaticAllocators, verboseLevel);
           delete allocList;

GeantGeoNavCollection::resize needs all three of these: it allocates
G4TouchableHandle / G4TouchableHistory out of the allocator pools, and
new G4Navigator needs the transportation and navigation-history state. By the
time the Runner calls it, all of them are gone.

Confirming experiment

One-line diagnostic change, gated on an environment variable so the same binary
tests both branches:

void GeantSetup::RMDeleter::operator()(G4RunManager* rm) const
{
    CELER_LOG(debug) << "Clearing Geant4 state";
    if (std::getenv("CELER_LEAK_RUN_MANAGER")) { return; }   // added
    delete rm;
}

That suppresses only the delete of the run manager. RunTermination() is
still called, and nothing else changes.

Result, one process per test exactly as ctest runs them:

test	stock	run manager leaked
LArSphereGeneratorTest.primary	SIGSEGV, rc=139	PASSED, rc=0
LArSphereGeneratorTest.direct	SIGSEGV, rc=139	PASSED, rc=0
LArSphereGeneratorTest.offload	SIGBUS, rc=138	PASSED, rc=0
DuneGeneratorTest.offload	SIGSEGV, rc=139	PASSED, rc=0
WlsGeneratorTest.primary	SIGSEGV, rc=139	PASSED, rc=0

All five pass, including every reference-value assertion (for
WlsGeneratorTest.primary: 221248 steps, 20 step iterations, 155575 photons
from wls-generate, 65536 from primary-generate, and the full action-label
set). The test takes 276 ms. So the physics and the optical loop are fine; the
only thing wrong is the lifetime of the Geant4 run manager.

Note the tests must be run one per process. Running the whole binary in a single
process fails the 2nd through 5th on Celeritas's own
geant_launch_count == 0 guard in GeantSetup.cc:88, which is unrelated and is
presumably why celeritas_add_standalone_tests registers each as its own ctest.

Suggested fix

Give OpticalStandaloneLoaded ownership of the GeantSetup (e.g. a
std::shared_ptr<GeantSetup> alongside geant_geo) so the run manager outlives
the Runner that is still allocating Geant4-backed state. The non-optical
standalone_input path at StandaloneInput.cc:67 builds GeantSetup as a
local in the same way and is worth checking for the same defect.

5. Coverage

Both platforms named in the original report are now verified against pristine
upstream develop 8cd6d69: macOS arm64 (this machine) and x86_64 Linux (MIT
SubMIT, SLURM job 6060272). Neither run used any CCM patch, and both checked at
runtime that the freshly built libraries were the ones actually loaded.

Not covered: any GPU configuration, and any build with CELERITAS_DEBUG=ON.
A debug build would be worth doing, since the disabled CELER_EXPECT in
GeantGeoNavCollection::resize might have caught this earlier.

The same result on x86_64 Linux, with a different Geant4 and compiler

Repeated end to end on the MIT SubMIT cluster (SLURM job 6060272, worker node
submit30, AlmaLinux x86_64), same pristine worktree of upstream develop 8cd6d69,
same five phases. bin/celer-config there:

"build_type": "relwithdebinfo,shared",
"core_geo": "Geant4",
"core_rng": "xorwow",
"debug": false,
"geant4": "global-dynamic,multithreaded,shared,gdml",
"openmp": "event",
"real_type": "double",
"units": "CGS",
"versions": { "CLHEP": "2.4.7.1", "Geant4": "11.1.3" }
"version": "0.7.0-dev.496+8cd6d692b"

test	pristine	patch built in, env var OFF	run manager leaked
LArSphereGeneratorTest.primary	rc=139	rc=139	PASSED
LArSphereGeneratorTest.direct	rc=139	rc=139	PASSED
LArSphereGeneratorTest.offload	rc=139	rc=139	PASSED
DuneGeneratorTest.offload	rc=139	rc=139	PASSED
WlsGeneratorTest.primary	rc=136	rc=136	PASSED

geocel/g4_GeantGeo passes 65 tests there too. The crash output is identical to
macOS, ending at status: Celeritas optical setup complete.

The middle column is the control that matters: the diagnostic patch is compiled
into that binary, but with the environment variable unset it takes the normal
delete rm path and crashes exactly as before. So the difference in the last
column comes from suppressing the deletion, not from having rebuilt.

This also disposes of the "maybe it is your Geant4 build" caveat. The two
platforms differ in nearly everything that could plausibly matter:

	macOS arm64	Linux x86_64
compiler	Apple clang 17.0.0	GCC 11.4.0 (Spack)
Geant4	11.3.2	11.1.3
Geant4 TLS model	initial-exec	global-dynamic
CLHEP	bundled	2.4.7.1 external
OpenMP	disabled	event-level
CMake	4.1.0	3.27.9

Two Geant4 minor versions, two compilers, two thread-local storage models, two
C++ standard libraries: identical failure, identical fix. This is the ownership
bug, not a local toolchain artefact.

Remaining open question: if these tests are green in upstream CI, something in
that environment must be keeping the Geant4 globals alive (or reinitialising
them). Worth asking, since the defect in the ownership is unambiguous and now
platform-independent.

6. Correction to make in the issue thread

The original report said the crash happens with "the untouched upstream test."
That was true, but the evidence behind it at the time came from a build tree
carrying our patches. It is now genuinely verified against unpatched upstream
develop at 8cd6d69, in a clean environment. Worth saying so plainly, since
the follow-up comment already flagged the report as possibly user error.