feat(aws-cdk): detailed ECS service and container-level logs on cdk deployment failures by iankhou · Pull Request #1505 · aws/aws-cdk-cli

iankhou · 2026-05-12T22:49:22Z

Pending tests and potential cleanups with custom resource implementation. Pending dual-role implementation (allow using lookup role or deploy role).

Logs from running with https://github.com/iankhou/cdk-app-with-problems. There are several branches demonstrating different failure modes:

Runtime error inside docker container
Runtime error inside docker container, but no CloudWatch logs configured
Failed health checks
Failed to pull image (network err or does not exist)
Container out of memory

See the following file for log examples:
lcdk deploy results.md

Checklist

This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
- Release notes for the new version:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

github-actions · 2026-05-12T22:49:32Z

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

…gestions to turn on cw logging format changes

…xplorationSdk

Copilot

Pull request overview

This PR enhances CDK deployment diagnostics by enriching failed CloudFormation resource errors with best-effort, service-specific investigation results—initially focused on ECS services—so users can see stopped-task reasons and recent CloudWatch Logs directly in failure output.

Changes:

Add ECS resource investigation that gathers stopped task details, container image info, and recent CloudWatch Logs (with line caps).
Extend stack diagnosis to attach and format additional per-resource diagnostic context.
Wire an optional “exploration SDK” (preferably lookup-role) into deploy/diagnose flows and expand the SDK ECS surface area to support investigation calls.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
packages/@aws-cdk/toolkit-lib/test/api/diagnosing/resource-investigation.test.ts	Adds unit tests for ECS identifier parsing and ECS investigation behaviors (stopped tasks, logs, truncation).
packages/@aws-cdk/toolkit-lib/test/api/diagnosing/snapshots/diagnosis-formatting.test.ts.snap	Updates snapshot header text.
packages/@aws-cdk/toolkit-lib/test/api/aws-auth/snapshots/sdk-logger.test.ts.snap	Updates snapshot header text.
packages/@aws-cdk/toolkit-lib/lib/toolkit/toolkit.ts	Passes an exploration SDK provider into the diagnoser for toolkit-driven diagnosis.
packages/@aws-cdk/toolkit-lib/lib/api/diagnosing/stack-diagnoser.ts	Enhances errors with additional investigation context and adds fallback diagnosis behavior.
packages/@aws-cdk/toolkit-lib/lib/api/diagnosing/resource-investigation.ts	Implements ECS service investigation (stopped tasks + CloudWatch Logs deep links + log truncation).
packages/@aws-cdk/toolkit-lib/lib/api/diagnosing/diagnosis-formatting.ts	Renders `additionalContext` blocks beneath resource errors in the formatted diagnosis output.
packages/@aws-cdk/toolkit-lib/lib/api/deployments/deployments.ts	Provides a lookup-role-based exploration SDK provider for deployments.
packages/@aws-cdk/toolkit-lib/lib/api/aws-auth/sdk.ts	Adds ECS methods (DescribeServices/DescribeTasks/DescribeTaskDefinition) to the SDK wrapper.
packages/@aws-cdk/toolkit-lib/lib/actions/diagnose/index.ts	Extends `TracedResourceError` with optional `additionalContext` and defines its shape.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

rix0rrr · 2026-06-15T13:26:03Z

+  const taskDefInfo = await getTaskDefinitionInfo(ecs, taskDefinitionArn, debug);
+
+  if (taskDefInfo && taskDefInfo.images.length > 0) {
+    results.push({ source: 'Container Images', messages: taskDefInfo.images });
+  }
+
+  const logConfigs = taskDefInfo?.logConfigs ?? [];
+
+  if (logConfigs.length === 0) {


The AI is right!

rix0rrr · 2026-06-15T13:28:35Z

+  // eslint-disable-next-line @cdklabs/promiseall-no-unbounded-parallelism
+  const logResults = await Promise.all(logConfigs.map(cfg => fetchRecentLogs(cwl, cfg, region, stoppedTaskResult.taskIds, debug)));
+  let hasLogs = false;
+  for (const context of logResults) {
+    if (context) {
+      results.push(context);
+      hasLogs = true;
+    }
+  }
+
+  if (!hasLogs) {
+    results.push({
+      source: 'CloudWatch Logs',
+      messages: ['No application logs found (container may not have started). Check the stopped task reasons above for details.'],
+    });
+  }


The AI is right :)

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

rix0rrr · 2026-06-15T13:19:18Z

Nice! I can see how this is all extremely helpful to users!

Having said that, this error looks extremely messy. I think we need to reconsider this, because nobody will be able to read this properly:

❌  ImagePullFailureStack failed: DeploymentError: Resource updates failed:
 └─ ImagePullFailureStack
     └─ Service
         └─ Service  (AWS::ECS::Service ServiceD69D759B)
            🛑 Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'."
               (RequestToken: 176e3d00-eb70-f4dc-bf7d-b86e2712e3e7, HandlerErrorCode: GeneralServiceException)
             ├─ 📋 ECS Stopped Tasks:
             │  Task stopped: CannotPullContainerError: pull image manifest has been retried 7 time(s): failed to resolve ref 229816860325.dkr.ecr.us-east-1.amazonaws.com/cdk-hnb659fds-container-assets-229816860325-us-east-1:nonexistent-tag-that-does-not-exist: 229816860325.dkr.ecr.us-east-1.amazonaws.com/cdk-hnb659fds-container-assets-229816860325-us-east-1:nonexistent-tag-that-does-not-exist: not found
             │  (2 other failed task(s) not shown)
             │  🔗 https://us-east-1.console.aws.amazon.com/ecs/v2/clusters/ImagePullFailureStack-ClusterEB0386A7-QMfJI4JmhBga/services/ImagePullFailureStack-ServiceD69D759B-mBSl2MWtMql9/tasks?status=STOPPED&region=us-east-1
             ├─ 📋 Container Images:
             │  DefaultContainer: 229816860325.dkr.ecr.us-east-1.amazonaws.com/cdk-hnb659fds-container-assets-229816860325-us-east-1:nonexistent-tag-that-does-not-exist
             └─ 📋 CloudWatch Logs:
                No application logs found (container may not have started). Check the stopped task reasons above for details.

How about we do the same type of error reporting that we do for validation errors? There should be code for it somewhere, and it should look ~roughly like this I think:

And perhaps fewer emoji and headers? For example, do the "ECS Stopped Tasks"/"Container Images"/"CloudWatch Logs" headers really add something?

I think Task stopped: <reason> already says enough, so we don't need the caption.
CloudWatch Logs: if there aren't any logs also feels unnecessary. Just saying "No CloudWatch Logs found" there should be enough.
I like the links! Especially if there are more than one, I think those could have a same-line leader word. Let's prefix them with a word, like Tasks: and Logs:?
What is a user going to do with Container Images: ? Are they ever useful?

I like that you're giving an example for turning on CloudWatch logs, but it looks incorrect:

                No CloudWatch Logs configuration found. Enable logging to see container output on failure.
                Example (CDK):
                  new ecs.FargateService(this, "Service", {
                    taskDefinition,
                    // Logging is enabled by default with CDK.
                    // If you disabled it, re-enable with:
                    // taskDefinition.addContainer("app", { logging: ecs.LogDrivers.awsLogs({ streamPrefix: "app" }) })
                  });

Just the addContainer call would be enough, and "(CDK)" should be obvious? 😉

rix0rrr

Good stuff! See my other comment on formatting, nearly good to go this!

rix0rrr · 2026-06-15T13:24:09Z

+/**
+ * Maximum number of log lines included per CloudWatch Logs context block.
+ *
+ * The formatter renders the messages array verbatim, so this is the
+ * single user-visible cap.
+ */
+const MAX_LOG_LINES = 50;


From the top or from the bottom?

If there's more than 50, we should probably get the LAST 50 lines, not the FIRST 50 lines. Are we doing that?

rix0rrr · 2026-06-15T13:26:03Z

+  const taskDefInfo = await getTaskDefinitionInfo(ecs, taskDefinitionArn, debug);
+
+  if (taskDefInfo && taskDefInfo.images.length > 0) {
+    results.push({ source: 'Container Images', messages: taskDefInfo.images });
+  }
+
+  const logConfigs = taskDefInfo?.logConfigs ?? [];
+
+  if (logConfigs.length === 0) {


The AI is right!

rix0rrr · 2026-06-15T13:26:40Z

+        'No CloudWatch Logs configuration found. Enable logging to see container output on failure.',
+        'Example (CDK):',
+        '  taskDefinition.addContainer("app", {',
+        '    image: ecs.ContainerImage.fromRegistry("my-image"),',


Suggested change

' image: ecs.ContainerImage.fromRegistry("my-image"),',

' // ...',

rix0rrr · 2026-06-15T13:26:50Z

+      source: 'CloudWatch Logs',
+      messages: [
+        'No CloudWatch Logs configuration found. Enable logging to see container output on failure.',
+        'Example (CDK):',


Suggested change

'Example (CDK):',

'Example:',

rix0rrr · 2026-06-15T13:28:35Z

+  // eslint-disable-next-line @cdklabs/promiseall-no-unbounded-parallelism
+  const logResults = await Promise.all(logConfigs.map(cfg => fetchRecentLogs(cwl, cfg, region, stoppedTaskResult.taskIds, debug)));
+  let hasLogs = false;
+  for (const context of logResults) {
+    if (context) {
+      results.push(context);
+      hasLogs = true;
+    }
+  }
+
+  if (!hasLogs) {
+    results.push({
+      source: 'CloudWatch Logs',
+      messages: ['No application logs found (container may not have started). Check the stopped task reasons above for details.'],
+    });
+  }


The AI is right :)

iankhou temporarily deployed to automation May 12, 2026 22:49 — with GitHub Actions Inactive

iankhou had a problem deploying to automation May 12, 2026 22:49 — with GitHub Actions Failure

github-actions Bot added the p2 label May 12, 2026

aws-cdk-automation requested a review from a team May 12, 2026 22:49

iankhou temporarily deployed to no-approval May 12, 2026 22:49 — with GitHub Actions Inactive

iankhou temporarily deployed to run-tests May 12, 2026 22:54 — with GitHub Actions Inactive

iankhou temporarily deployed to run-tests May 20, 2026 13:06 — with GitHub Actions Inactive

rix0rrr and others added 5 commits May 20, 2026 11:08

WIP cloudwatch

cfae31f

Clarify with comments

ad13b7b

POC for ECS improved logs

0927619

wip

33321e2

use tree nodes for logging, get info for unhealthy ecs instances, sug…

2a8c6c1

…gestions to turn on cw logging format changes

iankhou force-pushed the iankhou-ecs-logs-poc branch from 5cb6bde to 2a8c6c1 Compare May 27, 2026 20:53

iankhou temporarily deployed to no-approval May 27, 2026 20:53 — with GitHub Actions Inactive

iankhou changed the title ~~Iankhou ecs logs poc~~ feat(ecs): detailed service and container-level logs on cdk deployment failures May 27, 2026

iankhou temporarily deployed to run-tests May 27, 2026 20:57 — with GitHub Actions Inactive

iankhou added 6 commits May 28, 2026 12:47

promise to avoid race condition for concurrent callers of additionalE…

66ae39e

…xplorationSdk

formatting and correctness fixes

542f0fc

show only latest failure when container fails to start

98e2bd8

Merge branch 'main' into iankhou-ecs-logs-poc

96a14ff

unit tests, config consolidation

f1a23cf

Merge branch 'main' into iankhou-ecs-logs-poc

e6f9ea6

Copilot AI reviewed Jun 6, 2026

View reviewed changes

iankhou and others added 2 commits June 6, 2026 09:20

Apply suggestions from code review

4e0ca7a

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fixes

7943e4d

rix0rrr reviewed Jun 15, 2026

View reviewed changes

	' image: ecs.ContainerImage.fromRegistry("my-image"),',
	' // ...',

Conversation

iankhou commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

github-actions Bot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency Review

Scanned Files

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

rix0rrr Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

rix0rrr Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rix0rrr commented Jun 15, 2026

Uh oh!

rix0rrr left a comment

Choose a reason for hiding this comment

Uh oh!

rix0rrr Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

rix0rrr Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

rix0rrr Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

rix0rrr Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

rix0rrr Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

iankhou commented May 12, 2026 •

edited

Loading

github-actions Bot commented May 12, 2026 •

edited

Loading