Repository navigation
Fix per-thread workspace overlap with user-supplied work[] (lwork > 0) - #36
Merged
Merged
Conversation
When the caller supplies work[] (lwork > 0), each factorization thread
allocates its private iwork/dwork from the TAIL of that stack in
p?gstrf_WorkInit(). But p?gstrf_WorkFree(), called by every thread when it
finishes, reset the whole TAIL (top2 = size), releasing the work arrays
of all threads, including ones that were still running. A thread that
started after another one had finished then got TAIL memory overlapping
a live thread's marker[]/dense[] arrays and overwrote them in its
ifill()/?fill() initialization. That corrupted the factorization
and produced spurious zero pivots, e.g. intermittent ctest failures in
p[cz]test_4_{10,19}_2_100000000_LA:
z_div: division by zero
pzgssvx:info=15, izero=0, n=19, nrhs=2, imat=9
ThreadSanitizer reported ifill/zfill in one thread racing with
snode_dfs/column_dfs in another on the same work[] addresses.
Count the threads that hold TAIL work arrays and reset the TAIL only
when the last one frees its arrays. Applied to all four precisions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017GsYX7J15S6uvtUoRMWYwi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Intermittent ctest failures (the 2 failures reported in #34, also seen without CBLAS and with
THREAD_API=OPENMP):Over 20 runs of the full suite, the failing tests were always
pctest/pztest_4_{10,19}_2_100000000_LA. That means 4 threads and a user-suppliedwork[](-l 100000000). 13 of the 20 runs had at least one failure.Root cause
When
lwork > 0, each factorization thread takes its privateiwork/dworkfrom the TAIL end of the user stack inp?gstrf_WorkInit(). Each thread callsp?gstrf_WorkFree()when it finishes, and that function reset the whole TAIL:This releases the work arrays of every thread, including threads that are still running. With small matrices one thread can finish before another has even started. The late thread then gets TAIL memory that overlaps a live thread's
marker[]/dense[], and itsifill()/?fill()initialization overwrites them. The corrupted state shows up as a false zero pivot.ThreadSanitizer confirms it:
ifill/zfillin one thread race withpzgstrf_snode_dfs/pzgstrf_column_dfsin another thread on the samework[]addresses.Fix
In all four precisions (
SRC/p{s,d,c,z}memory.c):p?gstrf_WorkInit()counts the threads holding TAIL work arrays (nwork_users), under the existing stack lock.p?gstrf_WorkFree()resets the TAIL only when that count drops to 0.p?gstrf_SetupSpace()and the refactorization path inp?gstrf_MemInit().Testing
Built locally with GCC; the system-BLAS builds also used the CI flags (
-Wall -Wextra -Wpedantic):THREAD_API=OPENMPifill/zfill)This adds no compiler warnings. TSan still reports races on shared L/U columns. Those come from the library's spin-wait synchronization on plain flags, which TSan can't see, and they are unrelated to this bug.
🤖 Generated with Claude Code
https://claude.ai/code/session_017GsYX7J15S6uvtUoRMWYwi
Generated by Claude Code