feat(test-optimization): correlate WebdriverIO tests with RUM - #10049
feat(test-optimization): correlate WebdriverIO tests with RUM#10049juan-fernandez wants to merge 12 commits into
Conversation
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 97318a9a55
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
BenchmarksBenchmark execution time: 2026-08-31 17:56:02 Comparing candidate commit 5a4323c in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2299 metrics, 11 unstable metrics.
|
97318a9 to
33751bd
Compare
Overall package sizeSelf size: 8.66 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.4.0 | 127.33 kB | 447.04 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## juan-fernandez/orchestrion-async-context-callbacks #10049 +/- ##
======================================================================================
+ Coverage 91.38% 97.25% +5.87%
======================================================================================
Files 882 996 +114
Lines 137878 152786 +14908
Branches 12110 12286 +176
======================================================================================
+ Hits 125994 148598 +22604
+ Misses 11884 4188 -7696 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
33751bd to
633880e
Compare
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![datadog-prod-us1-4[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
The RUM cleanup records only the WebdriverIO browser object. Tests that use more than one window or origin can keep an old correlation cookie and can link later RUM events to the wrong test.
🤖 Datadog Autotest · Commit 633880e · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 633880e8fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1b02272c73
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e03228f5c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
A test that uses one browser window for more than one origin leaves the old origin's correlation cookie. A later test can send RUM data with the prior test execution ID.
🤖 Datadog Autotest · Commit 8e03228 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
@codex review |
|
/datadog autotest review |
There was a problem hiding this comment.
More details
This PR adds WebdriverIO 9 RUM-to-test correlation for Mocha and Jasmine;
it detects RUM after navigation, sets the test cookie and browser tags,
and stops and flushes RUM after hooks and before retries.
The new commit tracks every visited origin and removes its cookie,
so it fixes the earlier stale-cookie issue; no issue remains open.
🤖 Datadog Autotest · Commit 265c53c · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 265c53c1e4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb94d6b7f4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1e4c5245ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab91b5d16d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4bc6b2022d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8f89af0bf3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f918bcc386
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| typeof browser.scriptAddPreloadScript !== 'function') return | ||
|
|
||
| try { | ||
| if (rumBrowserPreloadScripts.has(browser)) return |
There was a problem hiding this comment.
When a BiDi test enters a native WebdriverIO retry and then navigates, the retry path reapplies the new execution ID to the current windows, but this early return preserves the preload script whose serialized argument still contains the previous attempt's ID. The next document executes that stale preload before RUM initializes, so its initial RUM events are associated with the prior attempt even if post-navigation cookie setting later corrects the ID. Fresh evidence after the earlier retry fix is that the final retry test explicitly keeps scriptAddPreloadScript at one call while changing IDs and never loads a subsequent document; replace the preload when its execution ID changes and exercise the resulting document lifecycle.
AGENTS.md reference: AGENTS.md:L129-L129
Useful? React with 👍 / 👎.
| const hookName = context.arguments?.[7] | ||
| if (!isRumCleanupPending || (type === 'Hook' && hookName === 'afterEach')) return | ||
|
|
||
| context.rumCleanupGenerator = cleanupRumBrowsers |
There was a problem hiding this comment.
For a Jasmine suite without afterEach, this defers cleanup until the next test or hook wrapper has started, at which point the tracing start channel has already bound the new test's store. If the previous RUM session is active, cleanupRumWindow publishes getRumTestExecutionId(browser, true) during that cleanup, causing the Mocha plugin to mark the new test as RUM-active and attach browser metadata even if it never uses RUM. Run the guaranteed cleanup before binding the next test span or preserve the prior test context; the current regression stubs stopRumSession to return false, so it does not assert the real span metadata affected here.
AGENTS.md reference: AGENTS.md:L129-L129
Useful? React with 👍 / 👎.
| } else if (type === 'Hook' && hookName === 'afterEach') { | ||
| context.resolveGenerator = cleanupRumBrowsers | ||
| context.rejectGenerator = cleanupRumBrowsers |
There was a problem hiding this comment.
When a framework invokes two afterEach hook wrappers, such as hooks from nested suites or a shared helper plus the suite, the first wrapper's completion enters this branch and stops RUM, removes the preload, and deletes the cookie. A later afterEach that interacts with the same page then runs without correlation because the browser has already been removed from the active set and no navigation occurs to register it again. Fresh evidence after the earlier single-hook fix is that this final condition accepts every individual afterEach wrapper and has no signal that the current one is the last; move cleanup to a lifecycle after all such hooks have completed.
AGENTS.md reference: AGENTS.md:L128-L128
Useful? React with 👍 / 👎.
| rumBrowsers.add(browser) | ||
| rumCorrelationBrowsers.add(browser) | ||
| rumBrowserTestExecutionIds.set(browser, testExecutionId) |
There was a problem hiding this comment.
When a BiDi page loads the RUM SDK but its session is sampled out, this pre-navigation registration keeps the browser in rumCorrelationBrowsers even though post-navigation detection correctly reports isRumActive: false. Cleanup later executes stopRumSession, whose return value is based only on the presence of the SDK's stopSession method, and therefore calls getRumTestExecutionId(browser, true), incorrectly emitting test.is_rum_active=true and waiting for a nonexistent active session. Track whether detection actually found an active session rather than treating every preloaded browser as active; the sampled-out regression currently exercises only the classic path, not this BiDi sibling.
AGENTS.md reference: AGENTS.md:L128-L128
Useful? React with 👍 / 👎.
f918bcc to
dc64c5d
Compare
6d983fb to
48242b7
Compare
dc64c5d to
5a4323c
Compare
Use synchronous generators for WebdriverIO browser operations so async composition stays in the transformed framework wrapper. No-RUM navigation microbenchmark (2M iterations, 5 trials, two fresh processes): baseline 48.4-73.8 ms; generator path 7.4-7.9 ms; 84-89% faster.
48242b7 to
eab5edc
Compare
5a4323c to
d73b378
Compare
2 participants
What does this PR do?
Adds WebdriverIO 9 RUM session-to-test correlation for the Mocha and Jasmine adapters.
The instrumentation:
afterEachhook.afterEachhook runs.The browser operations use normal async functions that return promises. Existing Orchestrion completion callbacks wait
for those promises before the instrumented WebdriverIO function continues or settles.
Motivation
Bring WebdriverIO to parity with the existing Playwright and Cypress RUM/test correlation behavior without requiring a
WebdriverIO service or user configuration.
Additional Notes
This is the upper PR in a two-PR stack:
Both PRs are drafts. Merge #10080 before rebasing this PR onto
master.Verification:
./node_modules/.bin/mocha packages/datadog-instrumentations/test/helpers/rewriter/index.spec.js— 36 passing./node_modules/.bin/mocha packages/datadog-instrumentations/test/webdriverio.spec.js— 58 passing./node_modules/.bin/mocha --timeout 60000 integration-tests/webdriverio/webdriverio.test-optimization.spec.js— 146 passing, 24 pending