Skip to content

[Flaky] AOT Selftests: process dies mid-run at fixture 481/1459 with no TAP trailer and no not ok #1152

Description

What happened?

The AOT Selftests CI leg intermittently dies mid-run rather than reporting a failing fixture. This is a different signature from the two AOT flakes already tracked (#1037 ExitTr_Removed, #990 Issue487_SV2*), both of which run the corpus to completion and fail on an assertion.

Observed on PR #1149 (job 98975322325):

failing run re-run, identical commit
fixtures started (# Running:) 481 1459
# Total failures: trailer absent # Total failures: 0
not ok lines none none
exit code 1 0

The last output is # Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22; the process dies between fixtures, before # Running: ThreadSafe_RapidBackgroundSetState is printed. In the passing run that fixture is index 481, immediately after ConnectedAnimation_OrphanOnlyPassDoesNotCrash at 480, so the ordering is the same and the run simply stops there.

Why this matters beyond "a flake": the job fails closed here (exit 1), but the failure carries no diagnostic at all — no failing fixture name, no assertion text, no trailer. There are two materially different explanations and the log cannot distinguish them:

  1. the host really died between fixtures, or
  2. the host produced more output that was lost, since it ends its run with a teardown-free TerminateProcess (Residual: full selftest suite 0xC0000005 at final process exit (distinct from #647 cross-fixture poisoning) #680) and the job pipes stdout through Tee-Object, so anything still buffered at termination is discarded.

If it is (2), the same truncation could in principle drop a genuine not ok on a future run.

Not a watchdog kill: the death is 32.9s after the first fixture (20:34:30.74920:35:03.638), far inside the suite budget, and none of the StartHangWatchdog / per-fixture timeout paths emit their diagnostics.

ThreadSafe_RapidBackgroundSetState is not in SelfTestRunner.DefaultAotSkipPatterns and is a plain registry entry (SelfTestFixtureRegistry.cs:571, :2278).

Steps to reproduce

Intermittent — it reproduced once in ~10 AOT legs on this branch and did not reproduce on re-run of the identical commit.

  1. dotnet publish tests/Reactor.AppTests.Host -p:PublishAotInternal=true -p:Platform=x64 -r win-x64 -c Release -o <out>
  2. & <out>/Reactor.AppTests.Host.exe --self-test 2>&1 | Tee-Object -FilePath aot-selftest-output.tap
  3. Compare (Select-String '# Running: ' aot-selftest-output.tap).Count against 1459 and check for the # Total failures: trailer. A short count with no trailer is this bug.

A cheap mitigation regardless of root cause: have the AOT job assert the trailer is present, so a truncated stream is reported as "truncated" rather than as an unexplained exit 1.

Reactor version / commit

f390d8f (PR #1149; the crash is independent of that PR — its diff only touches tests/Reactor.PackagedTests/*.cs, which the AOT job never compiles, since Reactor.AppTests.Host.csproj has no reference to Reactor.PackagedTests)

Platform

x64

.NET SDK version

10.0.x (windows-latest GitHub-hosted runner)

Windows version

windows-latest GitHub-hosted runner

Windows App SDK version

As resolved by main at the commit above.

Logs / stack trace

# Running: ConnectedAnimation_OrphanOnlyPassDoesNotCrash
ok ConnectedAnimOrphanOnly_DestinationMounted
ok ConnectedAnimOrphanOnly_NothingStarted
ok ConnectedAnimOrphanOnly_SurvivesRepeatedRoundTrips
# Fixture time: ConnectedAnimation_OrphanOnlyPassDoesNotCrash 22
##[error]Process completed with exit code 1.

(Expected next line, per the passing run: # Running: ThreadSafe_RapidBackgroundSetState.)

Confirmation

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions