`aspire stop` hangs indefinitely in CI, causing ~half of our main-branch test failures over the past 2 weeks

[afscrome]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

aspire stop — invoked from our TypeScript AppHost test harness's cleanup step, after the test body has already completed successfully — hangs indefinitely instead of exiting, consuming CI's full 7-minute hang-dump budget before the process is force-killed. This causes an otherwise fully-passing test to report as a failure (exit code 7/134), and has been our single largest source of CI flakiness: we estimate this accounts for roughly half of our main branch test failures over the past two weeks, recurring across multiple, unrelated integrations (Meilisearch x2, Floci, RustFs — all Aspire CLI 13.5.0).

We added extra diagnostics/log-capture around our TypeScript AppHost tests in CommunityToolkit/Aspire#1607 specifically to get visibility into this, and it's what let us pin down the following.

Expected Behavior

aspire stop should either complete promptly or fail with a clear, bounded error — it should never hang indefinitely.

Steps To Reproduce

We don't have a minimal repro (this appears tied to a shutdown-timing race, so it's intermittent), but the consistent signature across every occurrence we've captured, right as the AppHost begins tearing down:

fail: Aspire.Hosting.Backchannel.AuxiliaryBackchannelService[0]
      Error handling client connection on auxiliary backchannel
      System.ObjectDisposedException: Cannot access a disposed object.
      Object name: 'IServiceProvider'.
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
         at Aspire.Hosting.Backchannel.AuxiliaryBackchannelService.HandleClientConnectionAsync(Socket clientSocket, CancellationToken stoppingToken) in .../AuxiliaryBackchannelService.cs:line 157

...followed by:

• The aspire stop CLI process going completely silent, with no further log lines at all.
• Confirmed (via our CI's hang-dump watchdog output identifying the still-alive process tree, and per-invocation CLI log files captured by Suspect argument on FileNotFoundException in DistributedApplicationOptions.ResolveAssembly() #1607) that this specific aspire stop process is the one left hanging.
• The process eventually being SIGKILLed after the ~7 minute hang-dump timeout; attempts to take a memory dump of it fail with Microsoft.Diagnostics.NETCore.Client.ServerNotAvailableException: Unable to connect to Process <pid>, consistent with it being blocked on I/O rather than executing.
• In every case, aspire wait <resource> had already completed successfully and the test body itself had passed — only the cleanup call to aspire stop hangs.

Example job with captured logs: https://github.com/CommunityToolkit/Aspire/actions/runs/35761857913/job/106861834507 (Meilisearch), artifact ts-app-host-logs-Hosting.Meilisearch.Tests-ubuntu-latest shows the CLI log going silent immediately after this exception.

Reading AuxiliaryBackchannelService.HandleClientConnectionAsync (current main), we believe the mechanism is:

private async Task HandleClientConnectionAsync(Socket clientSocket, CancellationToken stoppingToken)
{
    try
    {
        ...
        var rpcTarget = new AuxiliaryBackchannelRpcTarget(
            serviceProvider.GetRequiredService<ILogger<AuxiliaryBackchannelRpcTarget>>(), // throws ObjectDisposedException here during shutdown
            ...);
        using var stream = new NetworkStream(clientSocket, ownsSocket: true); // never reached
        ...
    }
    catch (Exception ex)
    {
        logger.LogError(ex, "Error handling client connection on auxiliary backchannel"); // logs and returns — clientSocket never disposed
    }
}

Connections are dispatched from the accept loop as untracked fire-and-forget tasks (_ = Task.Run(...)), so the AppHost's DI container can be disposed mid-shutdown while a handler is still resolving services from it. When that throws, execution never reaches the using var stream = new NetworkStream(clientSocket, ownsSocket: true) that would own/dispose the socket — so the accepted socket appears to be leaked, and the client (aspire stop, mid handshake) is left blocked reading from a connection that will never send data or close.

Exceptions (if any)

See log excerpt above.

Anything else?

• Adam Ratzman (@adamint)'s Bound AppHost auxiliary backchannel handshake #19832 ("Bound AppHost auxiliary backchannel handshake", fixing aspire run --start-debug-session hangs while stopping an orphaned AppHost #19269) adds a client-side 10-second deadline around this handshake — a good mitigation for the symptom (a caller like aspire stop/aspire ps will no longer hang forever), and would likely have prevented our CI failures from consuming the full hang-dump window. However, it doesn't fix the underlying problem: aspire stop would still time out and fail to actually stop the AppHost cleanly in this scenario — it converts a silent hang into a bounded failure, but aspire stop still doesn't do its job. The server-side socket leak / DI-disposal race above looks unaddressed on main as of this writing.
• Bound AppHost auxiliary backchannel handshake #19832 is not in any released version yet (present on main/unreleased release/14.0 only; absent from release/13.5, which is what we currently run in CI at 13.5.0–13.5.4).
• Aspire CLI: 13.5.0 (also appears present on current main)
• Happy to help validate a fix against our downstream repro pattern if useful.

[joperezr]

🤖 Aspire triage pass — 2026-09-22 18:48 UTC

Area: area-cli, area-hosting
Kind: bug
Repro: partial — see lines 15–37 of the issue body
Affected version(s): 13.5.0–13.5.4 (latest: 13.5.4)
Severity: S2 — core CLI shutdown can hang indefinitely across unrelated CI scenarios with no released bounded failure

Urgency: 🟡 vNext — recommend milestone 13.6
Reason: meaningful, actionable product bug for the active release train; no evidence confirms it was introduced in 13.5, so servicing is not appropriate

Suggested owner(s): James Newton-King (@JamesNK) (score 79), backup Adam Ratzman (@adamint) (score 53)
Evidence:

• JamesNK merged Use repository NuGet configuration for SDK dump restores #20062, Remove flaky duplicate child log path test #19538, Color statuses in aspire ps output #19369 in area-cli (last 90d)
• JamesNK closed [main] aspire doctor taking 15 secs to run #19016, Use resolved dotnet paths in aspire doctor #19003, [13.5] Trace logging on aspire doctor causes weird indentation #19017 as completed in area-cli
• adamint merged Avoid live CLI rendering when stdout is redirected #19673, Fix Java starter JDK and transport reliability #19635, Bound AppHost auxiliary backchannel handshake #19832 and closed [13.4] Aspire CLI Can't discriminate between app hosts if running concurrent DistributedApplicationTestingBuilder in same test project #17617 in area-cli
• CODEOWNERS lists James Newton-King (@JamesNK) for the validated src/Aspire.Hosting/** path

Confidence: medium
Needs human?: no

[danegsta]

I'm taking a look at this.