Skip to content

Report why the persistent-tasks probe failed instead of guessing - #390

Open
ChrisRackauckas-Claude wants to merge 2 commits into
JuliaTesting:masterfrom
ChrisRackauckas-Claude:report-precompile-failure-in-persistent-tasks
Open

Report why the persistent-tasks probe failed instead of guessing#390
ChrisRackauckas-Claude wants to merge 2 commits into
JuliaTesting:masterfrom
ChrisRackauckas-Claude:report-precompile-failure-in-persistent-tasks

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown

Problem

test_persistent_tasks spawns a wrapper package that usings the target package and, from inside precompilation, writes a done.log sentinel. If the subprocess exits without writing it, the check reports:

┌ Error: Unexpected error: /tmp/jl_XXXX/done.log was not created, but precompilation exited
Persistent tasks: Test Failed

and the test fails as though the package holds a persistent Task.

But the sentinel is also missing whenever the wrapper simply failed to precompile, which is a different problem with a different fix — and the reason is unrecoverable, because the child runs Pkg.precompile(; io = devnull).

This matters most for the retryable module is missing from the cache race. That failure only warns and exits 0, and the warning goes to io. With io = devnull the run looks like a clean exit with no sentinel, indistinguishable from a genuine persistent task.

I hit this downstream and it took a temporary patch to a CI job to recover the message. Once visible it was unambiguous:

34843.2 ms  ? jl_ji9KfebcbW
1 dependencies failed but may be precompilable after restarting julia
┌ jl_ji9KfebcbW
│  ┌ Warning: Module SciMLBaseDifferentiationInterfaceExt with build ID fafbfcfd-… is missing from the cache.
└  └ @ Base loading.jl:2643

done_log_written = false
exitcode = 0
termsignal = 0

Exit 0, no signal, no persistent task — a precompilation race. Before recovering that output I had ruled out a real persistent task, a native-library segfault, and memory exhaustion one at a time, because the check gives the same message for all of them.

Change

Capture the child's precompilation output to a file inside the wrapper directory and, when the sentinel is missing, report it together with the subprocess exit code and termination signal, and note that a precompilation failure is the likelier cause:

@error "Unexpected error: … This usually means the wrapper package failed to precompile, \
        rather than that $pkgname holds a persistent task." \
       exitcode termsignal precompilation

Behavior is otherwise unchanged: the probe still returns true, output is still hidden on the happy path, and the "currently precompiling" branch is untouched. This only affects what is reported on failure.

Note the child's own stderr was already forwarded, so hard precompilation errors were partly visible; what was lost is everything Pkg writes to io, which is exactly where the missing-from-cache warning lands.

Tests

New fixture test/pkgs/PersistentTasks/FailsToPrecompile that fails to precompile deterministically, plus a testset asserting the error names precompilation as the cause and carries exitcode and the captured output.

Full suite passes locally on Julia 1.12.6 — all 13 test files green, test_persistent_tasks.jl 12/12 (was 6).

`test_persistent_tasks` spawns a wrapper package that `using`s the target and
writes a `done.log` sentinel from inside precompilation. When that subprocess
exits without writing the sentinel, the check reports the package as holding a
persistent task -- but the far more common cause is that the wrapper simply
failed to precompile, and the reason was unrecoverable because the child ran
`Pkg.precompile(; io = devnull)`.

That matters most for the retryable "module is missing from the cache" race,
which only warns and exits 0. The warning goes to `io`, so with `devnull` the
run looks like a clean exit with no sentinel, indistinguishable from a real
persistent task. Diagnosing one such case downstream took a temporary patch to
a CI job to recover the message.

Capture the precompilation output to a file inside the wrapper directory and,
when the sentinel is missing, report it alongside the subprocess exit code and
termination signal, and say that a precompilation failure is the likely cause.
Output is still hidden on the happy path.
@codecov

codecov Bot commented Aug 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.80%. Comparing base (31bd36c) to head (7d10510).

Files with missing lines Patch % Lines
src/persistent_tasks.jl 88.88% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #390      +/-   ##
==========================================
+ Coverage   87.45%   87.80%   +0.35%     
==========================================
  Files          12       12              
  Lines         526      533       +7     
==========================================
+ Hits          460      468       +8     
+ Misses         66       65       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@lgoettgens

lgoettgens commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

@ChrisRackauckas @ChrisRackauckas-Claude could you please explain the differences between this PR and #389? From the description, both seem to try to fix the same issue, but with different patches.

You may also add comments to that PR with things that should be changed there with an explanation.

PatrickHaecker pushed a commit to PatrickHaecker/Aqua.jl that referenced this pull request Aug 3, 2026
## PR JuliaTesting#389 vs PR JuliaTesting#390

**JuliaTesting#389 (`more_robust_persistent_task`, this branch)** — Fixes the root cause: splits the wait into an *unbounded* load phase and a *`tmax`-bounded* shutdown phase, so dependency precompilation no longer counts toward the persistent-task verdict. Raises `tmax` 5→30. Reports a precompile failure by *throwing an error* (with captured stderr), not by mislabeling it a persistent task.

**JuliaTesting#390 (`report-precompile-failure-in-persistent-tasks`)** — Only reports *why* the probe failed: captures `Pkg.precompile`'s `io` output to a file and logs an `@error` with `exitcode`/`termsignal`, then returns `true`. Leaves the timing model and `tmax` untouched.

**Why we took over only `termsignal`:** JuliaTesting#389 already captures **stderr**, and Julia sends exceptions *and* warnings there (`Pkg.precompile`'s default is `io = stderr`). So JuliaTesting#390's separate `io`-to-file capture is redundant — the same information through a different door. The only genuinely additive detail was `termsignal`, which distinguishes a signal-killed subprocess from a normal nonzero exit.
@ChrisRackauckas

Copy link
Copy Markdown
Contributor

I didn't know #389 existed, and it's better

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants