[13.6] Community Toolkit tests timeout in StopAsync

[afscrome]

Description

After upgrading the CommunityToolkit/Aspire repo from Aspire 13.5.0 to 13.6.0 (staging, 13.6.0+34db30a7d3733229da64a403dfc4e4e4d7a1a43b), one of our DbGate hosting tests now fails consistently on ubuntu-latest. DistributedApplication.StopAsync() throws TaskCanceledException after roughly 30 seconds. The same test passed on 13.5.0.

I wouldn't expect StopAsync to throw a timeout/cancellation exception at all. Even if stopping a resource takes a long time, the caller shouldn't get a TaskCanceledException out of StopAsync.

Failed run: https://github.com/CommunityToolkit/Aspire/actions/runs/36237790581/job/108392716621 (PR CommunityToolkit/Aspire#1625)

Test

The test starts an app with a single DbGate container, waits only for endpoints to be allocated, then stops the app. So StopAsync is called while the container is most likely still starting.

[Fact]
[RequiresDocker]
public async Task AddDbGateWithDefaultsAddsUrlAnnotations()
{
    using var builder = TestDistributedApplicationBuilder.Create();

    var dbgate = builder.AddDbGate("dbgate");

    var tcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
    dbgate.OnResourceEndpointsAllocated((resource, @event, ct) =>
    {
        tcs.SetResult();
        return Task.CompletedTask;
    });

    var app = await builder.BuildAsync();
    await app.StartAsync();
    await tcs.Task;

    var urls = dbgate.Resource.Annotations.OfType<ResourceUrlAnnotation>();
    Assert.Single(urls, u => u.DisplayText == "DbGate Dashboard");

    await app.StopAsync(); // throws TaskCanceledException after ~30s on 13.6
}

AddDbGate adds a plain container resource (dbgate/dbgate image) with an HTTP endpoint. There's no health check and no WaitFor.

Failure

failed CommunityToolkit.Aspire.Hosting.DbGate.Tests.AddDbGateTests.AddDbGateWithDefaultsAddsUrlAnnotations (32s 801ms)
  System.Threading.Tasks.TaskCanceledException : A task was canceled.
    at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
    at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
    at Aspire.Hosting.DistributedApplication.StopAsync(CancellationToken cancellationToken) in /_/src/Aspire.Hosting/DistributedApplication.cs:497
    at CommunityToolkit.Aspire.Hosting.DbGate.Tests.AddDbGateTests.AddDbGateWithDefaultsAddsUrlAnnotations() in /home/runner/work/Aspire/Aspire/tests/CommunityToolkit.Aspire.Hosting.DbGate.Tests/AddDbGateTests.cs:600

Expected behavior

StopAsync completes (as it did on 13.5.0), even if a container is still starting when it's called.

Environment

• Aspire.Hosting 13.6.0 (staging, 34db30a7d3733229da64a403dfc4e4e4d7a1a43b); previously passing on 13.5.0
• .NET SDK 10.0.401
• GitHub Actions ubuntu-latest (Ubuntu 24.04), Docker 28.0.4

[joperezr]

🤖 Aspire triage pass — 2026-09-26 19:19 UTC

Area: �rea-app-testing, �rea-hosting
Kind: regression
Repro: yes-minimal — see lines 13–38 of the issue body
Affected version(s): 13.6.0 staging (last working: 13.5.0; latest: 13.5.4)
Severity: S3 — shutdown during container startup fails, while waiting for readiness is a likely workaround

Urgency: 🟡 vNext — recommend milestone 13.6
Reason: meaningful, actionable regression in the active release train; it is not a shipped 13.5 servicing regression

Suggested owner(s): David Negstad (@danegsta) (score 3), backup Sébastien Ros (@sebastienros) (score 2)
Evidence:

• danegsta closed aspire stop hangs indefinitely in CI, causing ~half of our main-branch test failures over the past 2 weeks #20316 as completed in �rea-hosting (last 7d)
• sebastienros closed aspire run of a TS apphost runs even if there are compile errors #16645 as completed in �rea-app-testing (last 180d)
• CODEOWNERS lists David Negstad (@danegsta) for src/Aspire.Hosting/** and Sébastien Ros (@sebastienros) for src/Aspire.Hosting.Testing/**

Confidence: medium
Needs human?: no

[afscrome]

Just seen this on another Toolkit test after the 13.6 upgrade - https://github.com/CommunityToolkit/Aspire/actions/runs/36267589094/job/108475199225#step:6:585

[danegsta]

I'll look at digging into this next week, but it'll probably have to wait until after any remaining 13.6 dogfooding if it's transient.

[afscrome]

DB gate tests (specifically CommunityToolkit.Aspire.Hosting.DbGate.Tests.AddDbGateTests.AddDbGateWithDefaultsAddsUrlAnnotations) has failed consistently on every single PR run so far after the 13.6 upgrade. Dbx I've only seen one failure, but I mentioned it as it's the same error

Community Toolkit Upgrade Branch Runs: https://github.com/CommunityToolkit/Aspire/actions/workflows/dotnet-ci.yml?query=branch%3Acodex%2Faspire-13-6-upgrade

Failures:

1. https://github.com/CommunityToolkit/Aspire/actions/runs/36236014638/job/108387923098#step:6:2952
2. https://github.com/CommunityToolkit/Aspire/actions/runs/36237154201/job/108390986100#step:6:1215
3. https://github.com/CommunityToolkit/Aspire/actions/runs/36237790581/job/108392716621#step:6:1222
4. https://github.com/CommunityToolkit/Aspire/actions/runs/36262466321/job/108460868886#step:6:1283
5. https://github.com/CommunityToolkit/Aspire/actions/runs/36266375645/job/108471801169#step:6:1207
6. https://github.com/CommunityToolkit/Aspire/actions/runs/36267589094/job/108475199344#step:6:1738

One run did fail with a slightly different failure, but it was a bad attempt to fix the bug, so failed before the test got as far as calling StopAsync - https://github.com/CommunityToolkit/Aspire/actions/runs/36238708287/job/108428424142#step:6:6223

[afscrome]

I should also clarify - whilst it fails consistently in CI, I haven't been able to reproduce this failure locally.

[afscrome]

I have been able to mitigate this error by increasing the host shutdown timeout from 30 --> 90 secs. This suggests there's some kind of performance regression in the host shutdown process.

        builder.Services.Configure<HostOptions>(options =>
        {
            options.ShutdownTimeout = TimeSpan.FromSeconds(90);
        });

https://github.com/CommunityToolkit/Aspire/actions/runs/36315383963/job/108609082099?pr=1625

[danegsta]

I have been able to mitigate this error by increasing the host shutdown timeout from 30 --> 90 secs. This suggests there's some kind of performance regression in the host shutdown process.

        builder.Services.Configure<HostOptions>(options =>
        {
            options.ShutdownTimeout = TimeSpan.FromSeconds(90);
        });

https://github.com/CommunityToolkit/Aspire/actions/runs/36315383963/job/108609082099?pr=1625

I'll focus on treating this as a profiling run if it's not strictly a flakey failing test issue. I did fix a legitimate shutdown bug in DCP (microsoft/dcp#275) that's in the Aspire build from this release. It specifically introduced an extra synchronization step around emitting watch events to the controllers from API writes, so while it should be an extra lock around relatively fast operations, I'll double check that assumption.