Refactor Test Optimization API transport

[anmarchenko]

What

Remove the unused impacted-tests and related copied CI Visibility code, move the Test Optimization API transport under internal/testoptimization/api, integrate durations into that transport layer, and add focused coverage for the Test Optimization package.

Why

This reduces copied dead code from dd-trace-go, makes the Test Optimization API surface easier to reason about, and locks in behavior with tests before the next refactoring step.

E2E testing

Run the full ddtest run with real Datadog backend in a feature branch with local commits. Make sure that git upload and tests skipping works.

[anmarchenko]

E2E Test Report: SUCCESS

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

• Method: Local Shepherd/Crook testing
• Target: EU real backend (--target eu, datadoghq.eu), not mockdog
• Revision tested: 63be5da122b14509b127002a72b7268a820e52c3 (anmarchenko/remove-impacted-tests-dead-code)
• Primary passing run: vagrant, run ID 20260617-143012
• Additional TIA skip evidence: sidekiq, run ID 20260617-142719

Results

Check	Status	Evidence
ddtest PR branch built and executed	PASS	ddtest built from anmarchenko/remove-impacted-tests-dead-code and completed the Vagrant run
Real backend settings transport	PASS	/api/v2/libraries/tests/services/setting returned 200; require_git changed from true to false after git upload
Git metadata upload	PASS	search_commits returned 200; packfile returned 204
Skippable tests transport	PASS	/api/v2/ci/tests/skippable returned 200
Test suite durations transport	PASS	/api/v2/ci/ddtest/test_suite_durations returned 200; Vagrant cache included 480 RSpec suite durations and weights for 477 files
Parallel ddtest execution	PASS	Vagrant ran 477 test files across 4 local workers and passed
Test/coverage intake	PASS	citestcycle and citestcov uploads returned 202
TIA skipping behavior	PASS	Sidekiq loaded 24 skippables and skipped 4 tests; telemetry included test.itr.tests_skipping.count => 4.0

Primary Run Summary

./bin/crook run vagrant -c ddtest --dep ddtest=anmarchenko/remove-impacted-tests-dead-code --target eu --debug

• Service: vagrant
• Branch: anmarchenko/ddtest-pr87-e2e-vagrant
• Commit under test workload: 5bad49a2036e3714c959eed2ab08656a49d569d9
• Result: passed
• Files run: 477
• Workers: 4
• Duration: 2m42.855s

Issues Found

No PR-blocking issues found.

I also ran Sidekiq to exercise positive TIA skipping. The first Sidekiq attempt failed because Redis was not running locally. After starting Redis, the backend transport and skipping behavior worked, but the Sidekiq test command still failed due playground/environment test failures unrelated to this transport refactor. I used the clean Vagrant pass as the primary E2E result and the Sidekiq run only as additional evidence that the skippable-tests path can drive actual skips.

Verification

Verified against the real EU Datadog backend through API/intake responses and debug telemetry. Manual Datadog UI inspection was not performed in this run.

Test Methodology

1. Reviewed the PR transport refactor and compared endpoint/payload shapes with tracer implementations.
2. Refreshed tracer source clones with scripts/sync-tracers.py.
3. Created disposable local playground commits to exercise git upload from a feature branch.
4. Ran full ddtest through Crook against the EU backend with --debug.
5. Audited settings, git upload, skippables, suite durations, test/coverage intake, runner cache, and final run summaries.

---

This E2E test was performed by Shepherd - autonomous QA agent for Datadog Test Optimization.