Skip to content

Fix per-thread workspace overlap with user-supplied work[] (lwork > 0) - #36

Merged
xiaoyeli merged 1 commit into
masterfrom
claude/nifty-noether-m4f0vh
Oct 5, 2026
Merged

xiaoyeli merged 1 commit into
masterfrom
claude/nifty-noether-m4f0vh

Conversation

@xiaoyeli

@xiaoyeli xiaoyeli commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Problem

Intermittent ctest failures (the 2 failures reported in #34, also seen without CBLAS and with THREAD_API=OPENMP):

z_div: division by zero
** First factor: info       18, equed 0
   pzgssvx:info=15, izero=0, n=19, nrhs=2, imat=9, nfail=0

Over 20 runs of the full suite, the failing tests were always pctest/pztest _4_{10,19}_2_100000000_LA. That means 4 threads and a user-supplied work[] (-l 100000000). 13 of the 20 runs had at least one failure.

Root cause

When lwork > 0, each factorization thread takes its private iwork/dwork from the TAIL end of the user stack in p?gstrf_WorkInit(). Each thread calls p?gstrf_WorkFree() when it finishes, and that function reset the whole TAIL:

stack.used -= (stack.size - stack.top2);
stack.top2 = stack.size;

This releases the work arrays of every thread, including threads that are still running. With small matrices one thread can finish before another has even started. The late thread then gets TAIL memory that overlaps a live thread's marker[]/dense[], and its ifill()/?fill() initialization overwrites them. The corrupted state shows up as a false zero pivot.

ThreadSanitizer confirms it: ifill/zfill in one thread race with pzgstrf_snode_dfs/pzgstrf_column_dfs in another thread on the same work[] addresses.

Fix

In all four precisions (SRC/p{s,d,c,z}memory.c):

  • p?gstrf_WorkInit() counts the threads holding TAIL work arrays (nwork_users), under the existing stack lock.
  • p?gstrf_WorkFree() resets the TAIL only when that count drops to 0.
  • The counter is reset wherever the user stack is reinitialized: p?gstrf_SetupSpace() and the refactorization path in p?gstrf_MemInit().

Testing

Built locally with GCC; the system-BLAS builds also used the CI flags (-Wall -Wextra -Wpedantic):

Configuration Before After
Full suite, internal CBLAS, PTHREAD 13 of 20 runs had failures 0 of 30
The 4 flaky tests, looped intermittent 0 of 100
Full suite, system BLAS, PTHREAD — 0 of 20
Full suite, system BLAS, THREAD_API=OPENMP about 1 run in 6 failed 0 of 20
TSan: races in workspace setup (ifill/zfill) 30 0

This adds no compiler warnings. TSan still reports races on shared L/U columns. Those come from the library's spin-wait synchronization on plain flags, which TSan can't see, and they are unrelated to this bug.

🤖 Generated with Claude Code

https://claude.ai/code/session_017GsYX7J15S6uvtUoRMWYwi


Generated by Claude Code

When the caller supplies work[] (lwork > 0), each factorization thread
allocates its private iwork/dwork from the TAIL of that stack in
p?gstrf_WorkInit(). But p?gstrf_WorkFree(), called by every thread when it
finishes, reset the whole TAIL (top2 = size), releasing the work arrays
of all threads, including ones that were still running. A thread that
started after another one had finished then got TAIL memory overlapping
a live thread's marker[]/dense[] arrays and overwrote them in its
ifill()/?fill() initialization. That corrupted the factorization
and produced spurious zero pivots, e.g. intermittent ctest failures in
p[cz]test_4_{10,19}_2_100000000_LA:

  z_div: division by zero
  pzgssvx:info=15, izero=0, n=19, nrhs=2, imat=9

ThreadSanitizer reported ifill/zfill in one thread racing with
snode_dfs/column_dfs in another on the same work[] addresses.

Count the threads that hold TAIL work arrays and reset the TAIL only
when the last one frees its arrays. Applied to all four precisions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017GsYX7J15S6uvtUoRMWYwi
@xiaoyeli
xiaoyeli merged commit 1e4bc16 into master Oct 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants