feat(data-pipeline): add compression option for agentless export#2205
Open
paullegranddc wants to merge 2 commits into
Open
feat(data-pipeline): add compression option for agentless export#2205paullegranddc wants to merge 2 commits into
paullegranddc wants to merge 2 commits into
Conversation
# Motivation Since we are sending data to the intake directly, we should be compressing requests as they can be multi MB. The compression uses the zstd crate, which was already used in trace-utils. It links to a C library, so it is not usable everyhwere (WASM for instance) and adds ~500KB of artifact size. For these reasons, it is hidden behind a feature flag and should be enabled as required (implementation using the agentless option mostly). Another option would be rustzstd, an full rust port. Tge library is smaller (300KB) but it is notably slower.. (2x to 5x slower, with worse compression ratios at the same level) # Changes * Add a compression strategy parameter to send_with_retry * implement zstd compression * since compressio is implemented lower down, remove compression option from SendData * Add a "compression" feature flag on data-pipeline to be used in languages where it is needed
Contributor
📚 Documentation Check Results📦
|
Contributor
Clippy Allow Annotation ReportComparing clippy allow annotations between branches:
Summary by Rule
Annotation Counts by File
Annotation Stats by Crate
About This ReportThis report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality. |
Contributor
🔒 Cargo Deny Results📦
|
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 70a0d4d48a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| payload: Vec<u8>, | ||
| headers: &HeaderMap, | ||
| retry_strategy: &RetryStrategy, | ||
| compression_strategy: CompressionStrategy, |
There was a problem hiding this comment.
Adding compression_strategy here leaves the doctest immediately above with the old five-argument call, so cargo test --doc -p libdd-trace-utils --no-default-features now fails with E0061 for send_with_retry(&capabilities, &target, payload, &headers, &retry_strategy). Please pass CompressionStrategy::None (or another strategy) in the example so the required doc-test validation stays green.
Useful? React with 👍 / 👎.
| payload, | ||
| &headers, | ||
| &self.retry_strategy, | ||
| compression_strategy, |
There was a problem hiding this comment.
When a caller uses SendDataBuilder::with_compression(CompressionStrategy::Zstd { .. }), this new argument moves compression inside send_with_retry, but send_payload has already computed payload_len from the uncompressed Vec. As a result successful compressed SendData requests record SendDataResult.bytes_sent/telemetry as the pre-compression size even though the field is documented as bytes in the payload sent and previously measured the compressed final_payload; return the final payload length or compress before recording the metric.
Useful? React with 👍 / 👎.
|
|
||
| match send_with_retry(capabilities, &target, json_body, &headers, &retry_strategy).await { | ||
| #[cfg(feature = "compression")] | ||
| let compression_strategy = CompressionStrategy::Zstd { level: 1 }; |
There was a problem hiding this comment.
When the compression feature is enabled (for example in --all-features CI runs), this branch zstd-compresses the agentless request body, but test_agentless_export_body_shape in trace_exporter/mod.rs still configures httpmock with JSON body_includes matchers. I verified cargo test -p libdd-data-pipeline --all-features test_agentless_export_body_shape returns the mock 404 because the compressed bytes no longer match, so update the test to decode/disable compression or match on headers instead.
Useful? React with 👍 / 👎.
| test-utils = [] | ||
| regex-lite = ["libdd-common/regex-lite"] | ||
| # Enable zstd compression for the agentless trace intake sender. | ||
| compression = ["libdd-trace-utils/compression"] |
There was a problem hiding this comment.
Defining the feature only on libdd-data-pipeline leaves no way for the existing FFI release path to enable it: I checked the Cargo feature wiring and libdd-data-pipeline-ffi does not forward libdd-data-pipeline/compression, while libdd-profiling-ffi only exposes data-pipeline-ffi. In languages consuming the generated FFI artifacts, enabling data-pipeline support will therefore still build uncompressed agentless sends unless a pass-through feature is added.
Useful? React with 👍 / 👎.
Contributor
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
yannham
approved these changes
Jul 8, 2026
yannham
left a comment
yannham
left a comment
Contributor
There was a problem hiding this comment.
Beside some Codex comments that look relevant, the rest LGTM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Since we are sending data to the intake directly, we should be compressing requests as they can be multi MB.
The compression uses the zstd crate, which was already used in trace-utils.
It links to a C library, so it is not usable everyhwere (WASM for instance) and adds ~500KB of artifact size. For these reasons, it is hidden behind a feature flag and should be enabled as required (implementation using the agentless option mostly).
Another option would be rustzstd, an full rust port. Tge library is smaller (300KB) but it is notably slower.. (2x to 5x slower, with worse compression ratios at the same level)
Changes