Report why the persistent-tasks probe failed instead of guessing - #390
Open
ChrisRackauckas-Claude wants to merge 2 commits into
Conversation
`test_persistent_tasks` spawns a wrapper package that `using`s the target and writes a `done.log` sentinel from inside precompilation. When that subprocess exits without writing the sentinel, the check reports the package as holding a persistent task -- but the far more common cause is that the wrapper simply failed to precompile, and the reason was unrecoverable because the child ran `Pkg.precompile(; io = devnull)`. That matters most for the retryable "module is missing from the cache" race, which only warns and exits 0. The warning goes to `io`, so with `devnull` the run looks like a clean exit with no sentinel, indistinguishable from a real persistent task. Diagnosing one such case downstream took a temporary patch to a CI job to recover the message. Capture the precompilation output to a file inside the wrapper directory and, when the sentinel is missing, report it alongside the subprocess exit code and termination signal, and say that a precompilation failure is the likely cause. Output is still hidden on the happy path.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #390 +/- ##
==========================================
+ Coverage 87.45% 87.80% +0.35%
==========================================
Files 12 12
Lines 526 533 +7
==========================================
+ Hits 460 468 +8
+ Misses 66 65 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Collaborator
|
@ChrisRackauckas @ChrisRackauckas-Claude could you please explain the differences between this PR and #389? From the description, both seem to try to fix the same issue, but with different patches. You may also add comments to that PR with things that should be changed there with an explanation. |
PatrickHaecker
pushed a commit
to PatrickHaecker/Aqua.jl
that referenced
this pull request
Aug 3, 2026
## PR JuliaTesting#389 vs PR JuliaTesting#390 **JuliaTesting#389 (`more_robust_persistent_task`, this branch)** — Fixes the root cause: splits the wait into an *unbounded* load phase and a *`tmax`-bounded* shutdown phase, so dependency precompilation no longer counts toward the persistent-task verdict. Raises `tmax` 5→30. Reports a precompile failure by *throwing an error* (with captured stderr), not by mislabeling it a persistent task. **JuliaTesting#390 (`report-precompile-failure-in-persistent-tasks`)** — Only reports *why* the probe failed: captures `Pkg.precompile`'s `io` output to a file and logs an `@error` with `exitcode`/`termsignal`, then returns `true`. Leaves the timing model and `tmax` untouched. **Why we took over only `termsignal`:** JuliaTesting#389 already captures **stderr**, and Julia sends exceptions *and* warnings there (`Pkg.precompile`'s default is `io = stderr`). So JuliaTesting#390's separate `io`-to-file capture is redundant — the same information through a different door. The only genuinely additive detail was `termsignal`, which distinguishes a signal-killed subprocess from a normal nonzero exit.
Contributor
|
I didn't know #389 existed, and it's better |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
test_persistent_tasksspawns a wrapper package thatusings the target package and, from inside precompilation, writes adone.logsentinel. If the subprocess exits without writing it, the check reports:and the test fails as though the package holds a persistent
Task.But the sentinel is also missing whenever the wrapper simply failed to precompile, which is a different problem with a different fix — and the reason is unrecoverable, because the child runs
Pkg.precompile(; io = devnull).This matters most for the retryable
module is missing from the cacherace. That failure only warns and exits 0, and the warning goes toio. Withio = devnullthe run looks like a clean exit with no sentinel, indistinguishable from a genuine persistent task.I hit this downstream and it took a temporary patch to a CI job to recover the message. Once visible it was unambiguous:
Exit 0, no signal, no persistent task — a precompilation race. Before recovering that output I had ruled out a real persistent task, a native-library segfault, and memory exhaustion one at a time, because the check gives the same message for all of them.
Change
Capture the child's precompilation output to a file inside the wrapper directory and, when the sentinel is missing, report it together with the subprocess exit code and termination signal, and note that a precompilation failure is the likelier cause:
Behavior is otherwise unchanged: the probe still returns
true, output is still hidden on the happy path, and the "currently precompiling" branch is untouched. This only affects what is reported on failure.Note the child's own stderr was already forwarded, so hard precompilation errors were partly visible; what was lost is everything Pkg writes to
io, which is exactly where the missing-from-cache warning lands.Tests
New fixture
test/pkgs/PersistentTasks/FailsToPrecompilethat fails to precompile deterministically, plus a testset asserting the error names precompilation as the cause and carriesexitcodeand the captured output.Full suite passes locally on Julia 1.12.6 — all 13 test files green,
test_persistent_tasks.jl12/12 (was 6).