fix(test-optimization): enrich payload error telemetry by juan-fernandez · Pull Request #10045 · DataDog/dd-trace-js

juan-fernandez · 2026-08-28T11:52:44Z

What does this PR do?

Enriches Test Optimization payload failure telemetry with HTTP status codes and concrete network or delivery lifecycle error types. The same request-error and dropped-payload metrics now cover test-cycle, code-coverage, and Dynamic Instrumentation log payloads.

Motivation

Payload delivery failures were grouped too broadly, which made it difficult to distinguish backend responses from network, timeout, backpressure, and queue-limit failures during high-volume test sessions.

Additional Notes

This is a draft stacked on #10044 because it consumes the delivery lifecycle error codes introduced there.

Verification:

21 focused Test Optimization telemetry and writer tests pass
Changed-path coverage: 93.43% statements, 80.35% branches, 95.30% lines
npm run lint passes

dd-octo-sts · 2026-08-28T11:53:31Z

Overall package size

Self size: 8.43 MB
Deduped: 9.09 MB
No deduping: 9.09 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 445.14 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

datadog-official · 2026-08-28T11:55:40Z

⚠️ Warnings

⚠️ Your PR has warnings. Please review the issues below.

🚦 11 Pipeline jobs failed

Test Optimization | Test Optimization / integration (node-active) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

3 failed tests related to automatic log submission without a valid API key. AssertionError: The input did not match the regular expression /Hello simple log!/.

Test Optimization | Test Optimization / integration (node-latest) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

3 failed tests due to assertion errors: logs not submitted when required environment variable is defined without corresponding API key.

Test Optimization | Test Optimization / integration (node-maintenance) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

1 failed test. Assertion Error: The input did not match the regular expression /Hello simple log!/ in integration-tests/ci-visibility/automatic-log-submission.spec.js:290:16

View all 11 failed jobs.

📋 Copy fix prompt

CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Before you start, set up the Datadog software-delivery tooling so you can
query the CI data yourself:

1. Check whether you already have the Datadog software-delivery MCP tools
   (e.g. a `search_datadog_ci_pipeline_events` tool) and the `unblock-pr` skill.
2. If either is missing, STOP and ask me for permission before installing
   anything. Do not install or run anything until I have said yes.
3. Only with my explicit approval, set up the Datadog software-delivery MCP
   server and skills by following:
     https://docs.datadoghq.com/getting_started/software_delivery_mcp_tools/
   then restart so the skill is picked up.
4. If I decline, skip all of the above and work from the context below alone.

Then run /unblock-pr — it will pull the CI data itself. The job context below is what we already know.

If /unblock-pr is not available — because I declined the setup above, or it did not install — work from the context below instead.

Datadog has already classified this failure as caused by changes in this PR.
Take that as given and work the fix:

1. Locate the change. Diff this branch against its base and find the change
   that produces this error. Explain the mechanism, don't just name a file:
     git fetch origin && git diff $(git merge-base origin/juan-fernandez/test-optimization-delivery-lifecycle HEAD)...HEAD
2. Reproduce it locally. Run the failing job's command or test before
   proposing anything.
3. Propose the smallest fix that addresses the root cause — not a workaround,
   not a broadened assertion, not a disabled or skipped test.
4. Re-run the same command to confirm, and say exactly what you ran.
5. If the failure turns out to be intermittent rather than deterministic, say
   so plainly instead of "fixing" it — that is a flaky test, and patching it
   hides the problem.

If the right move is to re-run the job rather than change code, use the job
link in the context below. For GitHub Actions: `gh run rerun <run-id> --failed`,
where the run ID is the number after `/runs/` in that URL (not the trailing
number, which is the job ID).

Branch: juan-fernandez/test-optimization-error-telemetry

Test Optimization | Test Optimization / integration (node-active)
Commit: 88c69444833c5df058b20a230dc0e9daecdff07c
Error (code / test):
3 failed tests related to automatic log submission without a valid API key. AssertionError: The input did not match the regular expression /Hello simple log!/.
CI job: https://github.com/DataDog/dd-trace-js/actions/runs/33175821662/job/98867145277

Test Optimization | Test Optimization / integration (node-latest)
Commit: 88c69444833c5df058b20a230dc0e9daecdff07c
Error (code / test):
3 failed tests due to assertion errors: logs not submitted when required environment variable is defined without corresponding API key.
CI job: https://github.com/DataDog/dd-trace-js/actions/runs/33175821662/job/98867145483

Test Optimization | Test Optimization / integration (node-maintenance)
Commit: 88c69444833c5df058b20a230dc0e9daecdff07c
Error (code / test):
1 failed test. Assertion Error: The input did not match the regular expression /Hello simple log!/ in integration-tests/ci-visibility/automatic-log-submission.spec.js:290:16
CI job: https://github.com/DataDog/dd-trace-js/actions/runs/33175821662/job/98867145504

Plus 7 more failing jobs not shown here.

🧪 16 Tests failed in 1 job

Test Optimization | integration-playwright — ❌ 16 tests failed

View more details · View in GitHub Actions
❌ playwright@latest failure screenshots does not warn when the active transport can upload screenshots from integration-tests/playwright/playwright-reporting.spec.js
Expected values to be strictly equal:

0 !== 1
❌ playwright@latest failure screenshots warns when the active transport cannot upload screenshots from integration-tests/playwright/playwright-reporting.spec.js
Expected values to be strictly equal:

0 !== 1
❌ playwright@latest reporting via evp proxy can run and report tests from integration-tests/playwright/playwright-reporting.spec.js
gatherPayloadsUntilChildExit: child exited with no matching payloads (after 1000ms grace period)
❌ playwright@latest reporting via evp proxy error tags new version requests tags session and children with _dd.ci.library_configuration_error.known_tests when ... from integration-tests/playwright/playwright-reporting.spec.js
gatherPayloadsUntilChildExit: child exited with no matching payloads (after 1000ms grace period)
❌ playwright@latest reporting via evp proxy error tags new version requests tags session and children with _dd.ci.library_configuration_error.test_management_t... from integration-tests/playwright/playwright-reporting.spec.js
gatherPayloadsUntilChildExit: child exited with no matching payloads (after 1000ms grace period)
↳ and 11 more — View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog retried 17 tests - 1 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 98.49% (-0.08%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f876bff | Docs | View more details | Give us feedback!}

pr-commenter · 2026-08-28T12:02:12Z

Benchmarks

Benchmark execution time: 2026-08-28 13:42:39

Comparing candidate commit f876bff in PR branch juan-fernandez/test-optimization-error-telemetry with baseline commit ebf4b14 in branch juan-fernandez/test-optimization-delivery-lifecycle.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2306 metrics, 4 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:debugger-line-probe-with-snapshot-minimal-24

unstable max_rss_usage [-17.010MB; +10.548MB] or [-6.991%; +4.335%]

scenario:dogstatsd-with-tags-20

unstable cpu_user_time [-414.970ms; +271.235ms] or [-8.837%; +5.776%]
unstable execution_time [-413.922ms; +273.335ms] or [-8.672%; +5.726%]
unstable throughput [-98755.226op/s; +148917.447op/s] or [-5.611%; +8.461%]

codecov · 2026-08-28T12:22:21Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.49%. Comparing base (ebf4b14) to head (f876bff).

Additional details and impacted files

@@                                   Coverage Diff                                   @@
##           juan-fernandez/test-optimization-delivery-lifecycle   #10045      +/-   ##
=======================================================================================
+ Coverage                                                97.76%   98.49%   +0.72%     
=======================================================================================
  Files                                                      993      993              
  Lines                                                   149583   149599      +16     
  Branches                                                 12541    12797     +256     
=======================================================================================
+ Hits                                                    146243   147341    +1098     
+ Misses                                                    3340     2258    -1082

Flag	Coverage Δ
aiguard	`62.56% <ø> (-0.08%)`	⬇️
aiguard-integration	`59.15% <ø> (ø)`
apm-bucket-0	`62.00% <ø> (ø)`
apm-bucket-1	`67.57% <ø> (ø)`
apm-bucket-2	`67.14% <ø> (ø)`
apm-bucket-3	`63.80% <ø> (ø)`
apm-capabilities-tracing	`62.53% <100.00%> (+<0.01%)`	⬆️
apm-integrations-aerospike	`59.82% <ø> (ø)`
apm-integrations-confluentinc-kafka-javascript	`66.09% <ø> (-0.04%)`	⬇️
apm-integrations-couchbase	`60.36% <ø> (ø)`
apm-integrations-http	`65.80% <ø> (ø)`
apm-integrations-kafkajs	`66.73% <ø> (ø)`
apm-integrations-next	`63.32% <ø> (ø)`
apm-integrations-prisma	`61.42% <ø> (ø)`
appsec	`76.60% <ø> (ø)`
appsec-express_fastify_graphql	`73.04% <ø> (ø)`
appsec-integration	`49.25% <0.00%> (+<0.01%)`	⬆️
appsec-kafka_ldapjs_lodash	`67.08% <ø> (ø)`
appsec-mongodb-core_mongoose_mysql	`70.39% <ø> (ø)`
appsec-next	`56.65% <0.00%> (-0.01%)`	⬇️
appsec-node-serialize_passport_postgres	`69.84% <ø> (ø)`
appsec-sourcing_stripe_template	`68.22% <ø> (-0.01%)`	⬇️
debugger	`68.97% <ø> (+<0.01%)`	⬆️
instrumentations-bucket-0	`55.14% <ø> (ø)`
instrumentations-bucket-1	`63.66% <ø> (ø)`
instrumentations-bucket-10	`64.89% <ø> (ø)`
instrumentations-bucket-11	`65.55% <ø> (ø)`
instrumentations-bucket-12	`55.02% <ø> (ø)`
instrumentations-bucket-13	`55.31% <ø> (ø)`
instrumentations-bucket-14	`55.14% <ø> (ø)`
instrumentations-bucket-2	`56.23% <ø> (ø)`
instrumentations-bucket-3	`57.05% <ø> (ø)`
instrumentations-bucket-4	`63.11% <ø> (ø)`
instrumentations-bucket-5	`49.34% <0.00%> (-0.01%)`	⬇️
instrumentations-bucket-6	`65.00% <ø> (ø)`
instrumentations-bucket-7	`55.28% <ø> (ø)`
instrumentations-bucket-8	`62.48% <ø> (ø)`
instrumentations-bucket-9	`57.87% <0.00%> (-0.01%)`	⬇️
instrumentations-instrumentation-couchbase	`53.68% <ø> (ø)`
instrumentations-integration-esbuild	`34.42% <0.00%> (-0.01%)`	⬇️
llmobs-ai_anthropic_bedrock	`66.37% <ø> (ø)`
llmobs-bucket-1	`64.48% <ø> (ø)`
llmobs-openai	`66.35% <ø> (ø)`
llmobs-openai-agents_vertex-ai	`63.54% <ø> (+<0.01%)`	⬆️
llmobs-sdk	`75.79% <ø> (ø)`
master-coverage	`?`
openfeature	`59.77% <ø> (ø)`
openfeature-unit	`57.51% <ø> (ø)`
platform-core_esbuild_instrumentations-misc	`40.55% <0.00%> (-0.01%)`	⬇️
platform-integration	`64.50% <ø> (ø)`
platform-shimmer_unit-guardrails_webpack	`38.28% <0.00%> (-0.01%)`	⬇️
plugins-browser-bunyan_bullmq_cassandra	`65.79% <ø> (ø)`
plugins-bucket-0	`60.41% <ø> (?)`
plugins-bucket-1	`57.68% <ø> (-0.05%)`	⬇️
plugins-bucket-11	`66.28% <ø> (ø)`
plugins-bucket-18	`65.80% <ø> (ø)`
plugins-bucket-19	`64.41% <ø> (-0.02%)`	⬇️
plugins-bucket-20	`66.28% <ø> (ø)`
plugins-bucket-4	`60.60% <ø> (ø)`
plugins-cookie_cookie-parser_crypto	`54.69% <ø> (ø)`
plugins-fastify_fetch_fs	`64.94% <ø> (-0.04%)`	⬇️
plugins-generic-pool_google-cloud-pubsub_grpc	`68.57% <ø> (+0.01%)`	⬆️
plugins-handlebars_hapi_hono	`63.00% <ø> (ø)`
plugins-ioredis_knex_langgraph	`61.07% <ø> (ø)`
plugins-ldapjs_light-my-request_limitd-client	`62.73% <ø> (ø)`
plugins-lodash_mariadb_memcached	`63.35% <ø> (+0.09%)`	⬆️
plugins-moleculer_mongodb_mongodb-core	`65.65% <ø> (ø)`
plugins-mongoose_multer_mysql	`63.45% <ø> (ø)`
plugins-mysql2_nats_node-serialize	`65.64% <ø> (ø)`
plugins-opensearch_passport-http_pino	`63.47% <ø> (ø)`
plugins-postgres_process_pug	`62.56% <ø> (ø)`
plugins-redis_router_sequelize	`66.24% <ø> (ø)`
plugins-test-and-upstream-rhea_undici_url	`65.59% <ø> (ø)`
plugins-valkey_vm_winston	`61.82% <ø> (-0.05%)`	⬇️
plugins-ws	`63.87% <ø> (ø)`
profiling	`65.89% <ø> (ø)`
serverless-aws-sdk-aws-sdk	`54.83% <0.00%> (-0.01%)`	⬇️
serverless-aws-sdk-base-inject-field	`54.27% <ø> (ø)`
serverless-aws-sdk-bedrockruntime	`57.56% <ø> (ø)`
serverless-aws-sdk-client	`59.40% <ø> (ø)`
serverless-aws-sdk-dynamodb	`58.51% <ø> (ø)`
serverless-aws-sdk-eventbridge	`57.08% <0.00%> (-0.01%)`	⬇️
serverless-aws-sdk-kinesis	`62.60% <ø> (ø)`
serverless-aws-sdk-lambda	`60.51% <ø> (ø)`
serverless-aws-sdk-s3	`58.58% <ø> (ø)`
serverless-aws-sdk-serverless-peer-service	`63.18% <ø> (ø)`
serverless-aws-sdk-sns	`63.51% <ø> (ø)`
serverless-aws-sdk-sqs	`63.98% <ø> (ø)`
serverless-aws-sdk-stepfunctions	`58.38% <ø> (ø)`
serverless-aws-sdk-util	`54.94% <ø> (ø)`
serverless-bucket-0	`57.13% <ø> (ø)`
serverless-bucket-1	`63.14% <ø> (ø)`
test-optimization-cucumber	`70.45% <92.85%> (+<0.01%)`	⬆️
test-optimization-cypress	`64.63% <50.00%> (-0.01%)`	⬇️
test-optimization-jest	`72.12% <92.85%> (+0.13%)`	⬆️
test-optimization-mocha	`71.94% <92.85%> (+<0.01%)`	⬆️
test-optimization-playwright-playwright-atr	`59.00% <28.57%> (-0.01%)`	⬇️
test-optimization-playwright-playwright-efd	`60.18% <50.00%> (-0.01%)`	⬇️
test-optimization-playwright-playwright-final-status	`59.79% <28.57%> (-0.01%)`	⬇️
test-optimization-playwright-playwright-impacted-tests	`59.90% <28.57%> (-0.01%)`	⬇️
test-optimization-playwright-playwright-reporting	`60.67% <50.00%> (-0.01%)`	⬇️
test-optimization-playwright-playwright-test-management	`60.97% <50.00%> (-0.01%)`	⬇️
test-optimization-playwright-playwright-test-span	`59.13% <28.57%> (-0.05%)`	⬇️
test-optimization-selenium	`58.60% <28.57%> (-0.01%)`	⬇️
test-optimization-testopt	`?`
test-optimization-vitest	`72.53% <53.57%> (?)`
test-optimization-vitest-browser	`58.46% <28.57%> (-0.01%)`	⬇️
test-optimization-webdriverio	`65.06% <53.57%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

juan-fernandez · 2026-08-31T11:55:59Z

Consolidated into #10044 so the delivery lifecycle and error telemetry changes can merge as one PR. #10069 is now stacked directly on #10044.

juan-fernandez added the semver-patch label Aug 28, 2026

juan-fernandez force-pushed the juan-fernandez/test-optimization-error-telemetry branch 3 times, most recently from a465cbe to 047676b Compare August 28, 2026 13:10

fix(test-optimization): enrich payload error telemetry

f876bff

juan-fernandez force-pushed the juan-fernandez/test-optimization-error-telemetry branch from 047676b to f876bff Compare August 28, 2026 13:31

This was referenced Aug 31, 2026

fix(test-optimization): retry payload delivery failures #10069

Draft

fix(test-optimization): bound payload delivery lifecycle #10044

Draft

juan-fernandez closed this Aug 31, 2026

Navigation Menu

Uh oh!

fix(test-optimization): enrich payload error telemetry - #10045

fix(test-optimization): enrich payload error telemetry#10045
juan-fernandez wants to merge 1 commit into
juan-fernandez/test-optimization-delivery-lifecyclefrom
juan-fernandez/test-optimization-error-telemetry

juan-fernandez commented Aug 28, 2026

Uh oh!

dd-octo-sts Bot commented Aug 28, 2026

Uh oh!

datadog-official Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Aug 28, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Unstable benchmarks

scenario:debugger-line-probe-with-snapshot-minimal-24

scenario:dogstatsd-with-tags-20

Uh oh!

codecov Bot commented Aug 28, 2026 •

edited

Loading

Uh oh!

juan-fernandez commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

juan-fernandez commented Aug 28, 2026

What does this PR do?

Motivation

Additional Notes

Uh oh!

dd-octo-sts Bot commented Aug 28, 2026

Overall package size

Uh oh!

datadog-official Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

pr-commenter Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

Unstable benchmarks

scenario:debugger-line-probe-with-snapshot-minimal-24

scenario:dogstatsd-with-tags-20

Uh oh!

codecov Bot commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

juan-fernandez commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

datadog-official Bot commented Aug 28, 2026 •

edited

Loading

pr-commenter Bot commented Aug 28, 2026 •

edited

Loading

codecov Bot commented Aug 28, 2026 •

edited

Loading