test(ffe): cover agentless CDN configuration-source contract by leoromanovsky · Pull Request #7277 · DataDog/system-tests

leoromanovsky · 2026-07-07T22:17:22Z

Motivation

Server SDK agentless flag delivery needs one shared system-test contract before the SDK implementations land. These tests are intentionally red behind missing_feature; they define the cross-SDK behavior we expect for configuration-source selection, CDN polling, cold failures, warm preservation, and no-Agent reachability.

Tracking: FFL-2731, under epic FFL-2662.

Internal RFC: Supporting OpenFeature Agentless mode in Server SDKs

Changes

Configuration-source tests

Adds parametric FFE tests for remote_config, explicit cdn, default cdn, customer endpoint override, invalid source, cold failure/recovery, warm preservation, timeout, and poller no-overlap.
Uses existing DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true to enable the current provider; the renamed contract under test is DD_FLAGGING_CONFIGURATION_SOURCE*.

Mock FFE CDN

Adds /mock/ufc/config with test-owned response timelines.
Reuses existing tests/parametric/test_ffe/flags-v1.json; malformed UFC stays inline.
Removes invented configuration-source request signals. Tests prove CDN by actual mock requests, path, auth, ETag, status codes, and OpenFeature evaluation.
Uses standard 500 for 5xx retryable failures. No 509.

Docker reachability

Adds host.docker.internal:host-gateway for the test agent and for parametric SDK containers when their env points at host.docker.internal.
Adds a live smoke test that fetches the mock from inside the test-agent container on the same Docker network.

Decisions

Mock scope
Keep the fixture FFE/UFC-specific for this PR. A generic HTTP fixture can come later if another suite needs it.

Configuration model
Customer-hosted HTTP endpoints are covered as cdn with a different base URL, not a fourth SDK mode.

Provider enablement
This suite does not rename provider enablement. It uses the current system-tests/SDK provider flag and focuses the new contract on configuration-source env vars.

SDK Gates

SDK / manifest	Implementation ticket	System-test ticket	Manifest reason
Java	`FFL-2693`	`FFL-2731`	Java agentless CDN implementation plus shared configuration-source contract
Go	`FFL-2695`	`FFL-2731`	Go agentless CDN implementation plus shared configuration-source contract
Node.js	`FFL-2697`	`FFL-2731`	Node.js agentless CDN implementation plus shared configuration-source contract
Python	`FFL-2699`	`FFL-2731`	Python agentless CDN implementation plus shared configuration-source contract
Ruby	`FFL-2701`	`FFL-2731`	Ruby agentless CDN implementation plus shared configuration-source contract
.NET	`FFL-2703`	`FFL-2731`	.NET agentless CDN implementation plus shared configuration-source contract
PHP	`FFL-2705`	`FFL-2731`	PHP agentless CDN implementation plus shared configuration-source contract
C++	`irrelevant`	`FFL-2731`	Not in scope for v1
Rust	`irrelevant`	`FFL-2731`	Not in scope for v1

Coverage

Source Selection

Scenario	Test	What it proves
Explicit RC baseline	`test_remote_config_positive_ignores_cdn_env`	RC applies `flags-v1.json`; CDN base URL can be present but mock CDN must receive zero requests.
Remote config absent	`test_remote_config_without_rc_does_not_fallback_to_cdn`	`remote_config` without RC stays default/not-ready and does not fall back to CDN.
Explicit CDN	`test_explicit_cdn_positive`	`DD_FLAGGING_CONFIGURATION_SOURCE=cdn` fetches the mock and evaluates the same flag value as RC.
Default CDN	`test_default_cdn_positive`	Omitted source selects CDN and does not require Agent RC.
Customer endpoint	`test_customer_http_endpoint_default_cdn_positive`	Exact endpoint base URL works as CDN delivery, proving customer-hosted HTTP is just a base URL override.
Invalid source	`test_invalid_configuration_source_fails_closed`	Unknown source does not silently fall back to CDN.

remote_config positive: Agent RC flags-v1.json -> value "green"; mock requests_total == 0
remote_config absent: no RC payload + CDN env present -> default/not-ready; mock requests_total == 0
explicit/default/customer CDN: valid -> HTTP 200 + flags-v1.json + ETag "ufc-v1" -> value "green"
invalid source: DD_FLAGGING_CONFIGURATION_SOURCE=invalid + CDN env present -> no mock request

Cold Failure And Recovery

Scenario	Test	What it proves
Missing auth cold	`test_missing_auth_cold`	Missing API key cannot make a cold provider ready.
Malformed cold	`test_malformed_cold`	HTTP 200 with malformed UFC cannot make a cold provider ready.
Request timeout cold	`test_request_timeout_cold`	A response slower than `DD_FLAGGING_CONFIGURATION_SOURCE_REQUEST_TIMEOUT_SECONDS` is treated as cold failure.
Bad-to-good retry	`test_bad_to_good_cold_recovery`	Initial 5xx does not permanently poison cold startup; later valid UFC recovers.
Bad-to-unchanged cold	`test_bad_to_unchanged_cold_preserves_not_ready`	5xx followed by 304 before any accepted UFC does not create fake last-known-good state.

missing auth: no DD_API_KEY -> HTTP 401 -> default/not-ready
malformed cold: malformed -> HTTP 200 + {"flags": [ -> default/not-ready
timeout cold: timeout -> sleeps 1.5s; SDK timeout=1s -> default/not-ready
bad-to-good: server_error -> valid == HTTP 500 -> HTTP 200 flags-v1.json -> value "green"
bad-to-unchanged: server_error -> not_modified == HTTP 500 -> HTTP 304 -> default/not-ready

Warm State Preservation

Scenario	Test	What it proves
Missing auth warm	`test_missing_auth_warm`	Auth failure after valid UFC preserves last-known-good evaluator state.
Good-to-bad	`test_good_to_bad_warm_preservation`	5xx after valid UFC preserves last-known-good evaluator state.
Good-to-unchanged	`test_good_to_unchanged_etag_sequence`	304 sends `If-None-Match` and keeps evaluator state unchanged.
Malformed warm	`test_malformed_warm`	Malformed UFC after valid UFC cannot corrupt last-known-good state.

missing auth warm: valid -> unauthorized == HTTP 200 flags-v1.json -> HTTP 401 -> value "green"
good-to-bad: valid -> server_error == HTTP 200 flags-v1.json -> HTTP 500 -> value "green"
good-to-unchanged: valid -> not_modified == HTTP 200 ETag "ufc-v1" -> HTTP 304 with If-None-Match "ufc-v1" -> value "green"
malformed warm: valid -> malformed == HTTP 200 flags-v1.json -> HTTP 200 malformed bytes -> value "green"

Poller And Fixture

Scenario	Test	What it proves
Delayed no-overlap	`test_delayed_no_overlap`	Poll interval shorter than response latency does not create concurrent CDN requests.
Metadata-only status	`test_mock_ffe_cdn_status_is_metadata_only`	Mock status exposes counters/request metadata only, not UFC bytes.
Docker reachability	`test_mock_ffe_cdn_reachable_from_test_agent_network`	The default mock URL is reachable from a Docker container on the test-agent network.

delayed no-overlap: delayed_valid -> sleeps 0.5s -> HTTP 200; poll interval=0.2s; max_in_flight == 1
reachability: test-agent container -> GET http://host.docker.internal:<port>/mock/ufc/config with DD-API-KEY -> HTTP 200; requests_total == 1

Validation

$ ./run.sh -S PARAMETRIC -L python --skip-parametric-build tests/parametric/test_ffe/test_mock_ffe_cdn_reachability.py -q
Scenario: PARAMETRIC
Library: python@4.10.7
1 passed in 5.14s

$ ./run.sh -S PARAMETRIC -L python --skip-parametric-build --collect-only -q tests/parametric/test_ffe/test_configuration_sources.py tests/parametric/test_ffe/test_mock_ffe_cdn_reachability.py
19 tests collected

System Tests (python, prod) / parametric / parametric (1)
  https://github.com/DataDog/system-tests/actions/runs/28916585866/job/85785079227

  report.json records:

  {
    "nodeid": "tests/parametric/test_ffe/
    test_mock_ffe_cdn_reachability.py::Test_Feature_Flag_Mock_FFE_CDN_Reachability::test_mock
    _ffe_cdn_reachable_from_test_agent_network",
    "outcome": "passed",
    "setup": "passed",
    "call": "passed",
    "teardown": "passed"
  }

github-actions · 2026-07-07T22:17:57Z

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/test_configuration_sources.py                 @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/test_the_test/test_mock_ffe_agentless_backend.py                  @DataDog/system-tests-core
utils/docker_fixtures/_mock_ffe_agentless_backend.py                    @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
utils/_features.py                                                      @DataDog/system-tests-core
utils/build/docker/dotnet/weblog/Controllers/FeatureFlagEvaluatorController.cs  @DataDog/apm-dotnet @DataDog/asm-dotnet @DataDog/system-tests-core
utils/build/docker/java/parametric/src/main/java/com/datadoghq/trace/controller/FeatureFlagEvaluatorController.java  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/java/spring-boot/src/main/java/com/datadoghq/system_tests/springboot/featureflag/FeatureFlagEvaluatorController.java  @DataDog/apm-java @DataDog/asm-java @DataDog/system-tests-core
utils/build/docker/nodejs/express/app.js                                @DataDog/dd-trace-js @DataDog/system-tests-core
utils/build/docker/python/parametric/apm_test_client/server.py          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/docker_fixtures/_core.py                                          @DataDog/system-tests-core
utils/docker_fixtures/_test_agent.py                                    @DataDog/system-tests-core
utils/docker_fixtures/_test_clients/_test_client_parametric.py          @DataDog/system-tests-core

datadog-datadog-prod-us1 · 2026-07-07T22:25:56Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 299f691 | Docs | Datadog PR Page | Give us feedback!}

- Add parametric FFE source-mode test class - Cover RC, CDN, default CDN, and negative fixture cases

- Add mock CDN control and metadata-only status endpoints - Add valid, malformed, and unchanged UFC fixtures

- Add source-mode class rows to all target SDK manifests - Gate rows pending agentless CDN source-mode implementations

- Register mock CDN fixture via pytest_plugins - Keep mock handler typing compatible with instance server state

- Add endpoint-form CDN base URL coverage - Add 509 and 304 transition cases for source-mode L3 - Assert CDN source metadata across mock-backed cases

- add deterministic retry/etag response sequences for CDN source-mode tests - expose metadata-only status code evidence and endpoint URL handling

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 81bbcdd41a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

cbeauchesne

I didn't looked closely the test logic, but as I know nothing aboutthe tested feature, a review from someone else will be a better idea :)

For the framework usage, I do have few questions, the most important one is the compatibility with all setup where system-tests runs. Then, just a few request change on the test-the-test stuff, and all good.

leoromanovsky · 2026-07-08T12:46:50Z

I didn't looked closely the test logic, but as I know nothing aboutthe tested feature, a review from someone else will be a better idea :)

For the framework usage, I do have few questions, the most important one is the compatibility with all setup where system-tests runs. Then, just a few request change on the test-the-test stuff, and all good.

@cbeauchesne Thanks for the suggestions; normal and expected that FFE will approve changes to features, I will hold the PR for their review.

cbeauchesne

AGTM for framwork usage. I let the real review for the test logic :)

sameerank

You may want to make further adjustments when enabling these tests against an SDK, but this looks like a good start and seems aligned with the RFC so far

sameerank · 2026-07-09T23:40:24Z

+
+    server = MockFFEAgentlessBackendServer(worker_id)
+    try:
+        env = {"DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL": server.library_config_url}


Don't forget to update the DD_FLAGGING_CONFIGURATION_SOURCE* references in the PR description. Also references to "cdn" are now "agentless"

Thank you; yes, performing a careful synchronization before merging:

RFC -> java trace, run against local update system test -> push trace -> push system test 🚀

sameerank · 2026-07-09T23:41:16Z

+    """Validate source selection for Agent RC, agentless, and customer endpoint overrides."""
+
+    @parametrize("library_env", [{"configuration_source": "remote_config", "response": "valid"}], indirect=True)
+    def test_remote_config_positive_ignores_agentless_env(


The PR description needs some updating:

test_remote_config_positive_ignores_cdn_env should be changed to this

there's a reference to tests/parametric/test_ffe/test_mock_ffe_cdn_reachability.py in this folder which is no longer in the diff

sameerank · 2026-07-10T00:06:23Z

+class Test_Feature_Flag_Configuration_Source_Mock_Fixture:
+    """Validate that the mock backend exposes only metadata needed by the tests."""
+
+    def test_mock_ffe_agentless_backend_status_is_metadata_only(


Your manifest file updates gate this entire file, e.g.

system-tests/manifests/java.yml

Line 3737 in 299f691

tests/parametric/test_ffe/test_configuration_sources.py: missing_feature (FFL-2693 tracks Java agentless configuration-source implementation; FFL-2731 tracks system-tests configuration-source contract)

But this at least looks like a test that is independent of the SDKs and doesn't need to be gated. Maybe this could set elsewhere like tests/test_the_test/

sameerank · 2026-07-10T00:18:17Z

+        test_library: APMLibrary,
+        mock_ffe_agentless_backend: MockFFEAgentlessBackendServer,
+    ) -> None:
+        mock_ffe_agentless_backend.reset()


Some feedback from Claude that might be helpful here:

mock_ffe_agentless_backend.reset() runs after the test_library container is already up. It's fine today because provider polling starts at ffe_start() (called later), but if any SDK begins agentless polling at process init rather than provider-set, an early erroneous request would be wiped by reset() and _assert_no_mock_requests would pass falsely. Capturing the clean baseline after ffe_start(), or documenting the lifecycle assumption, would harden it.

sameerank · 2026-07-10T00:23:01Z

+        self.send_response(status_code)
+        self.send_header("Content-Type", "application/json")
+        self.end_headers()


.send_response and .end_headers are wrapped in a block in _handle_config

system-tests/utils/docker_fixtures/_mock_ffe_agentless_backend.py

Lines 179 to 191 in 51372e9

try:

if response in {"delayed_valid", "timeout"}:

time.sleep(TIMEOUT_RESPONSE_SECONDS if response == "timeout" else DELAYED_RESPONSE_SECONDS)

status_code, body, headers = _response_for_response(

response=response,

has_auth=_has_auth(request_headers),

)

self.server.state.record_response(status_code)

self.send_response(status_code)

for key, value in headers.items():

self.send_header(key, value)

self.end_headers()

Would it make sense to do the same here?

sameerank · 2026-07-10T00:44:08Z

+            var activation = (
+                Environment.GetEnvironmentVariable("DD_FEATURE_FLAGS_ENABLED")
+                ?? Environment.GetEnvironmentVariable("DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED")
+                ?? "0").ToLowerInvariant();


This adds DD_FEATURE_FLAGS_ENABLED as an alias for the legacy enable flag, which makes sense for updating system-tests for using the new env variables.

But fyi, this is the e2e weblog controller, and the parametric suite this PR adds runs against utils/build/docker/dotnet/parametric/Endpoints/ApmTestApi.cs so this edit isn't touched by what the new tests run against

Dismissing at reviewer request after platform feedback was addressed.

leoromanovsky added 11 commits July 7, 2026 18:27

test(03-03): add failing source mode scaffold

ee5aa77

- Add parametric FFE source-mode test class - Cover RC, CDN, default CDN, and negative fixture cases

feat(03-03): add mocked CDN source-mode fixture

492c9c1

- Add mock CDN control and metadata-only status endpoints - Add valid, malformed, and unchanged UFC fixtures

test(03-03): gate source-mode system tests

fdc6516

- Add source-mode class rows to all target SDK manifests - Gate rows pending agentless CDN source-mode implementations

refactor(03-03): clean up source-mode scaffold imports

82e64ee

- Register mock CDN fixture via pytest_plugins - Keep mock handler typing compatible with instance server state

fix(03-03): tighten source-mode scaffold assertions

bbdccdd

fix(03-03): stabilize source-mode mock CDN checks

4448cfb

test(05-03): add failing source-mode sequence contract

8371fd2

- Add endpoint-form CDN base URL coverage - Add 509 and 304 transition cases for source-mode L3 - Assert CDN source metadata across mock-backed cases

feat(05-03): implement source-mode mock CDN evidence support

ca65538

- add deterministic retry/etag response sequences for CDN source-mode tests - expose metadata-only status code evidence and endpoint URL handling

test(06-01): harden source-mode red contract coverage

a69fc66

test(ffe): cover customer CDN endpoint source mode

92271fe

test(ffe): link source-mode manifests to FFL ticket

2874813

leoromanovsky force-pushed the gsd/03-03-source-modes-scaffold branch from 73df0cf to 2874813 Compare July 7, 2026 22:30

leoromanovsky added 5 commits July 7, 2026 18:47

test(ffe): reuse UFC fixture for source-mode CDN mock

d91fa37

test(ffe): configure CDN response sequences from tests

f6e9404

test(ffe): rename source modes to configuration sources

994bb07

test(ffe): annotate configuration-source manifests with SDK tickets

6ab9c61

test(ffe): use configuration source env vars

a9b81dd

leoromanovsky changed the title ~~test(ffe): cover agentless CDN source-mode contract~~ test(ffe): cover agentless CDN configuration-source contract Jul 7, 2026

leoromanovsky added 5 commits July 7, 2026 19:39

test(ffe): group configuration source scenarios

7e1a2e1

test(ffe): ignore configuration source suite for cpp and rust

4669f70

test(ffe): simplify mock fixture metadata

7ab6e88

test(ffe): keep mock status metadata only

9c93287

test(ffe): harden configuration-source contract

81bbcdd

leoromanovsky marked this pull request as ready for review July 8, 2026 04:38

leoromanovsky requested review from a team as code owners July 8, 2026 04:39

leoromanovsky requested review from Kyle-Verhoog, PerfectSlayer, cbeauchesne, claponcet, daniel-romano-DD, ncybul, pavlokhrebto, sameerank and xlamorlette-datadog and removed request for a team July 8, 2026 04:39

chatgpt-codex-connector Bot reviewed Jul 8, 2026

View reviewed changes

Comment thread tests/parametric/test_ffe/test_configuration_sources.py Outdated

Comment thread utils/docker_fixtures/_mock_ffe_cdn.py Outdated

Comment thread tests/parametric/test_ffe/test_configuration_sources.py Outdated

cbeauchesne previously requested changes Jul 8, 2026

View reviewed changes

Comment thread tests/parametric/test_ffe/test_mock_ffe_cdn_reachability.py Outdated

Comment thread tests/parametric/test_ffe/test_mock_ffe_cdn_reachability.py Outdated

Comment thread tests/parametric/test_ffe/test_configuration_sources.py Outdated

Comment thread utils/docker_fixtures/_core.py

test(ffe): address configuration-source review feedback

4b67703

leoromanovsky requested review from cbeauchesne and dd-oleksii and removed request for pavlokhrebto July 8, 2026 12:49

leoromanovsky added 3 commits July 8, 2026 21:01

Test feature flags agentless source

ff4b30a

Use server distribution route for agentless mock

51372e9

Re-gate Java agentless configuration tests

299f691

cbeauchesne reviewed Jul 9, 2026

View reviewed changes

sameerank approved these changes Jul 10, 2026

View reviewed changes

cbeauchesne approved these changes Jul 10, 2026

View reviewed changes

	try:
	if response in {"delayed_valid", "timeout"}:
	time.sleep(TIMEOUT_RESPONSE_SECONDS if response == "timeout" else DELAYED_RESPONSE_SECONDS)

	status_code, body, headers = _response_for_response(
	response=response,
	has_auth=_has_auth(request_headers),
	)
	self.server.state.record_response(status_code)
	self.send_response(status_code)
	for key, value in headers.items():
	self.send_header(key, value)
	self.end_headers()

Uh oh!

Conversation

leoromanovsky commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

SDK Gates

Coverage

Source Selection

Cold Failure And Recovery

Warm State Preservation

Poller And Fixture

Validation

Uh oh!

github-actions Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 7, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbeauchesne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leoromanovsky commented Jul 8, 2026

Uh oh!

cbeauchesne left a comment

Choose a reason for hiding this comment

Uh oh!

sameerank left a comment

Choose a reason for hiding this comment

Uh oh!

sameerank Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

leoromanovsky Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

sameerank Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

sameerank Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

sameerank Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

sameerank Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

sameerank Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leoromanovsky commented Jul 7, 2026 •

edited

Loading

github-actions Bot commented Jul 7, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 7, 2026 •

edited by datadog-official Bot

Loading