You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The AOT Selftests CI leg intermittently dies mid-run rather than reporting a failing fixture. This is a different signature from the two AOT flakes already tracked (#1037ExitTr_Removed, #990Issue487_SV2*), both of which run the corpus to completion and fail on an assertion.
The last output is # Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22; the process dies between fixtures, before # Running: ThreadSafe_RapidBackgroundSetState is printed. In the passing run that fixture is index 481, immediately after ConnectedAnimation_OrphanOnlyPassDoesNotCrash at 480, so the ordering is the same and the run simply stops there.
Why this matters beyond "a flake": the job fails closed here (exit 1), but the failure carries no diagnostic at all — no failing fixture name, no assertion text, no trailer. There are two materially different explanations and the log cannot distinguish them:
If it is (2), the same truncation could in principle drop a genuine not ok on a future run.
Not a watchdog kill: the death is 32.9s after the first fixture (20:34:30.749 → 20:35:03.638), far inside the suite budget, and none of the StartHangWatchdog / per-fixture timeout paths emit their diagnostics.
ThreadSafe_RapidBackgroundSetState is not in SelfTestRunner.DefaultAotSkipPatterns and is a plain registry entry (SelfTestFixtureRegistry.cs:571, :2278).
Steps to reproduce
Intermittent — it reproduced once in ~10 AOT legs on this branch and did not reproduce on re-run of the identical commit.
Compare (Select-String '# Running: ' aot-selftest-output.tap).Count against 1459 and check for the # Total failures: trailer. A short count with no trailer is this bug.
A cheap mitigation regardless of root cause: have the AOT job assert the trailer is present, so a truncated stream is reported as "truncated" rather than as an unexplained exit 1.
Reactor version / commit
f390d8f (PR #1149; the crash is independent of that PR — its diff only touches tests/Reactor.PackagedTests/*.cs, which the AOT job never compiles, since Reactor.AppTests.Host.csproj has no reference to Reactor.PackagedTests)
Platform
x64
.NET SDK version
10.0.x (windows-latest GitHub-hosted runner)
Windows version
windows-latest GitHub-hosted runner
Windows App SDK version
As resolved by main at the commit above.
Logs / stack trace
# Running: ConnectedAnimation_OrphanOnlyPassDoesNotCrash
ok ConnectedAnimOrphanOnly_DestinationMounted
ok ConnectedAnimOrphanOnly_NothingStarted
ok ConnectedAnimOrphanOnly_SurvivesRepeatedRoundTrips
# Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22
##[error]Process completed with exit code 1.
(Expected next line, per the passing run: # Running: ThreadSafe_RapidBackgroundSetState.)
What happened?
The
AOT SelftestsCI leg intermittently dies mid-run rather than reporting a failing fixture. This is a different signature from the two AOT flakes already tracked (#1037ExitTr_Removed, #990Issue487_SV2*), both of which run the corpus to completion and fail on an assertion.Observed on PR #1149 (job 98975322325):
# Running:)# Total failures:trailer# Total failures: 0not oklinesThe last output is
# Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22; the process dies between fixtures, before# Running: ThreadSafe_RapidBackgroundSetStateis printed. In the passing run that fixture is index 481, immediately afterConnectedAnimation_OrphanOnlyPassDoesNotCrashat 480, so the ordering is the same and the run simply stops there.Why this matters beyond "a flake": the job fails closed here (exit 1), but the failure carries no diagnostic at all — no failing fixture name, no assertion text, no trailer. There are two materially different explanations and the log cannot distinguish them:
TerminateProcess(Residual: full selftest suite 0xC0000005 at final process exit (distinct from #647 cross-fixture poisoning) #680) and the job pipes stdout throughTee-Object, so anything still buffered at termination is discarded.If it is (2), the same truncation could in principle drop a genuine
not okon a future run.Not a watchdog kill: the death is 32.9s after the first fixture (
20:34:30.749→20:35:03.638), far inside the suite budget, and none of theStartHangWatchdog/ per-fixture timeout paths emit their diagnostics.ThreadSafe_RapidBackgroundSetStateis not inSelfTestRunner.DefaultAotSkipPatternsand is a plain registry entry (SelfTestFixtureRegistry.cs:571,:2278).Steps to reproduce
Intermittent — it reproduced once in ~10 AOT legs on this branch and did not reproduce on re-run of the identical commit.
dotnet publish tests/Reactor.AppTests.Host -p:PublishAotInternal=true -p:Platform=x64 -r win-x64 -c Release -o <out>& <out>/Reactor.AppTests.Host.exe --self-test 2>&1 | Tee-Object -FilePath aot-selftest-output.tap(Select-String '# Running: ' aot-selftest-output.tap).Countagainst 1459 and check for the# Total failures:trailer. A short count with no trailer is this bug.A cheap mitigation regardless of root cause: have the AOT job assert the trailer is present, so a truncated stream is reported as "truncated" rather than as an unexplained exit 1.
Reactor version / commit
f390d8f (PR #1149; the crash is independent of that PR — its diff only touches
tests/Reactor.PackagedTests/*.cs, which the AOT job never compiles, sinceReactor.AppTests.Host.csprojhas no reference toReactor.PackagedTests)Platform
x64
.NET SDK version
10.0.x (
windows-latestGitHub-hosted runner)Windows version
windows-latest GitHub-hosted runner
Windows App SDK version
As resolved by
mainat the commit above.Logs / stack trace
(Expected next line, per the passing run:
# Running: ThreadSafe_RapidBackgroundSetState.)Confirmation
ExitTr_Removed+ 3 sibling exit-transition fixtures fail intermittently in the AOT Selftests leg (reproduces onmain) #1037 (AOT leg,ExitTr_Removed— assertion failure, full run) and [Bug] Issue487_SV2* fail deterministically on non-interactive sessions — InteractionTracker completes scrolls without moving #990 (Issue487_SV2*— deterministic on non-interactive sessions, full run); neither covers a mid-run process death with no trailer.mainbranch — the AOT host under test is built entirely frommainsources at this commit.