[SDTEST-3823] Add test discovery cache support

[anmarchenko]

What

Customers can speed up ddtest plan by restoring a previous discovery cache file and passing it with --test-discovery-cache <path> or DD_TEST_OPTIMIZATION_RUNNER_TEST_DISCOVERY_CACHE=<path>. When the cache is valid, ddtest reuses the previously discovered test list and skips Ruby full test discovery. After a successful full discovery, ddtest refreshes ./.testoptimization/tests-discovery/tests.json, so CI can persist that file and feed it back into the next run.

Why

If test files did not change, ddtest should not have to discover the same tests again.

E2E testing

Run these scenarios on a Ruby project with TIA enabled and at least one skippable test.

1. Cache creation and restore path:

   • Run ddtest plan without --test-discovery-cache.
   • Verify ./.testoptimization/tests-discovery/tests.json is created and ends with _ddtest_discovery_cache_metadata.
   • Persist that file through the CI cache, restore it on a later run, and pass the restored path with --test-discovery-cache <path> or DD_TEST_OPTIMIZATION_RUNNER_TEST_DISCOVERY_CACHE=<path>.

2. Cache hit:

   • Make a change outside the test root, for example under app/.
   • Run ddtest plan with the restored cache.
   • Verify the planner logs that cached test discovery is used, Ruby full discovery is not invoked, and the produced plan still applies TIA skips from the cached test list.

3. Test-root invalidation:

   • Modify, add, delete, rename, or leave untracked a file under the effective test root, for example spec/**/* for RSpec or test/**/* for Minitest.
   • Run ddtest plan with the restored cache.
   • Verify ddtest rejects the cache, runs full discovery, and writes fresh metadata to ./.testoptimization/tests-discovery/tests.json.

4. Settings and cache safety:

   • Re-run with a mismatched --tests-location, --tests-exclude-pattern, framework, platform, corrupt cache file, missing cache file, or cache whose source commit is unavailable locally.
   • Verify ddtest does not use the cache and falls back to full discovery or the existing fast file discovery fallback when full discovery fails.

5. Monorepo scope:

   • From a project subdirectory, create and restore a cache for that project.
   • Change a sibling project test file and verify the cache is still reused.
   • Change a test-root file in the current project and verify the cache is invalidated.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 89f0867366

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[anmarchenko]

E2E Test Report: SUCCESS

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

• Method: Local E2E planning runs with sidekiq via Shepherd crook
• Backend: local mockdog using ddtest-sidekiq-tm-disabled-tia-mix
• Command shape: crook run sidekiq -c ddtest-plan --dep ddtest=anmarchenko/test-discovery-cache --debug
• Revision tested: 7f3107e3a8fa240c633e6b83fe5b9bee62b46061

Results

Scenario	Result
Initial run without discovery cache	PASS: full discovery ran and wrote .testoptimization/tests-discovery/tests.json with _ddtest_discovery_cache_metadata
Same commit cache reuse, repeated	PASS: cache reused; Ruby full discovery command was not invoked; plan artifacts matched baseline
Later commit on same branch with only non-test changes	PASS: cache reused; plan artifacts matched baseline
Sibling branch with non-test-only changes	PASS: cache reused safely; plan artifacts matched baseline
Committed test-root change	PASS: cache rejected with test discovery file changed: test/actors_test.rb; full discovery ran
Untracked test-root file	PASS: cache rejected with the untracked test file path; full discovery ran
tests-location mismatch	PASS: cache rejected with tests location mismatch; scoped full discovery ran
Corrupt cache file	PASS: cache rejected with JSON parse error; full discovery ran
Missing cache file	PASS: import failure was non-fatal; full discovery ran
Missing source commit	PASS: cache rejected with source commit unavailable; full discovery ran
No-common-ancestor orphan branch, same test tree	PASS: cache reused; plan artifacts matched baseline
No-common-ancestor orphan branch, changed test tree	PASS: cache rejected with test discovery file changed: test/actors_test.rb; full discovery ran
Go unit test suite	PASS: go test ./... completed successfully after allowing module downloads

Baseline plan remained stable where expected:

• Test files discovered: 41
• Fully skipped files: 3
• Test files to run: 38
• Runners: 4
• Estimated time saved: 1.42%

Issues Found

No blocking issues found.

One behavior to call out for product review: cache validation is tree-diff based, not ancestry based. A no-common-ancestor orphan branch reused the cache when the effective test/ tree was identical to the cache source. When the orphan branch changed test/actors_test.rb, ddtest rejected the cache and ran full discovery. If unrelated histories should always invalidate, this PR needs an additional no-merge-base rejection. If identical test-root tree is sufficient, current behavior is consistent.

Minor logging note: cache-hit runs correctly log Cached test discovery succeeded, but later also print Full test discovery succeeded; using full discovery results.... Ruby discovery was not invoked, so this wording is a little confusing but did not affect behavior.

Verification

Datadog UI verification was not applicable for these runs: this was local mockdog plan-only validation of planner/cache behavior, so no real Datadog UI test session was produced.

Test Methodology

Created disposable commits and branches in the sidekiq playground to exercise git-aware cache validation, including same-commit reuse, same-branch non-test changes, sibling branches, no-common-ancestor orphan branches, test-root changes, untracked test files, settings mismatch, corrupt/missing cache input, and unavailable cache source commit. Verified cache decisions from debug logs and compared stashed .testoptimization plan artifacts under crook-data/runs.

---

This E2E test was performed by Shepherd - autonomous QA agent for Datadog Test Optimization