[lagunex]

Summary

The ADOT Java agent embeds async-profiler from v2.32.0 and exports profiles over OTLP. This PR lets the operator turn it on for every Java-instrumented pod. It is driven by a new java.profiling block in --auto-instrumentation-config, which the Helm chart sets in aws-observability/helm-charts#422.

Profiles go to the cloudwatch-agent agent's OTLP HTTP endpoint (the shared OTLP endpoint; today enabled with agent.config: default:otel or a custom http_endpoint), on the port that agent's config sets. The operator opens no receiver for profiling.

Nothing changes unless java.profiling.enabled is exactly "true". The agent's port and Service declarations (internal/manifests/collector) are untouched, so metrics, logs and traces behave as before.

Order and dependencies.

• This PR merges and ships in an operator release first; feat(chart): add experimental Java continuous profiling via the ADOT java agent aws-observability/helm-charts#422 follows.
• Profiles reach CloudWatch only with a CloudWatch agent that has the profiles pipeline: v1.300075.0 or later (feat(translator): generate a profiles pipeline from the otlp section amazon-cloudwatch-agent#2300). Without it, the injected profiler's exports are refused and the application is unaffected.

One commit: feat(instrumentation): enable the ADOT Java embedded profiler from java.profiling.

What

• main.go: setLangEnvVars copies java.profiling.enabled to AUTO_INSTRUMENTATION_JAVA_PROFILING_ENABLED, and every other java.profiling key, as JSON, to AUTO_INSTRUMENTATION_JAVA_PROFILING_SETTINGS. Other languages are ignored.
• pkg/instrumentation/javaprofiling.go (new) is called from the Java branch of sdkInjector.inject, after the javaagent is injected. For each instrumented Java container it:
  • sets OTEL_AWS_PROFILER_ENABLED=true and OTEL_AWS_PROFILER_ENDPOINT=http://cloudwatch-agent.amazon-cloudwatch:<port>/v1development/profiles.
    • <port> is the port of opentelemetry.collect.otlp.http_endpoint in the cloudwatch-agent AmazonCloudWatchAgent's config. It is read with the existing client, like the default Instrumentation's endpoints, so no new RBAC is needed.
    • The port is 4318 when the config has no usable http_endpoint.
    • The daemonset Service uses internalTrafficPolicy: Local, so profiles stay on the node.
  • adds one disk-backed emptyDir per pod, opentelemetry-profiler-data-java, mounted at /otel-profiler-data-java, and sets OTEL_AWS_PROFILER_DATA_DIR=/otel-profiler-data-java/<container-name>. Profiling then works with readOnlyRootFilesystem, and the files stay off the javaagent volume;
  • sizes the volume with sizeLimit = volume_size_limit once per profiled container, because each JVM keeps its own rotating JFR budget (128 MiB plus the current window). The default is 160Mi, in whole bytes and at least 16Mi; any other value is logged and the default is used;
  • maps every other key to OTEL_AWS_PROFILER_<KEY upper-cased>:
    • Reserved keys: enabled; endpoint (refused, as is any key containing it); data_dir (an absolute base path); volume_size_limit.
    • Keys must be letters, digits and underscores, and values must not contain control characters.
    • Keys that ADOT v2.32.0 does not read are still injected, and logged once.
  • lets a variable the container already sets win, whether it comes from env, valueFrom or a ConfigMap envFrom.
• Opt-out: a container that sets OTEL_AWS_PROFILER_ENABLED to anything but "true" is not profiled. No pod or namespace annotation is read.
• A container is also skipped, with a log line, in the cases below. The pod is never rejected.
  • The pod is a Windows pod (the kubernetes.io/os node selector or spec.os.name).
  • The pod runs on Fargate (schedulerName: fargate-scheduler or the eks.amazonaws.com/fargate-profile label), which has no node-local agent.
  • The container may use OTLP gRPC and does not set its own profiler endpoint: OTEL_EXPORTER_OTLP_PROTOCOL=grpc (from env or envFrom, any case), a protocol from valueFrom (unknown), or -Dotel.exporter.otlp.protocol=grpc in JAVA_TOOL_OPTIONS. See "Why gRPC containers are skipped".
  • It reads a Secret or prefixed envFrom, which the operator cannot resolve.
  • The pod already has a volume with the profiler volume name, or the container already mounts it or /otel-profiler-data-java.
• helper.go: defaultProfilerVolumeSize (160Mi).

Why gRPC containers are skipped

The profiler follows the application's OTLP protocol, and the operator points it at the agent's OTLP HTTP endpoint. In ADOT Java v2.32.0 (instrumentation/serviceevents/.../config/ServiceEventsConfig.java):

• line 351: profilerProtocol = parseProfilerProtocol(getConfigValue("OTEL_EXPORTER_OTLP_PROTOCOL")). Profiles have no per-signal protocol variable.
• lines 484-501, getConfigValue: the system property otel.exporter.otlp.protocol is checked before the environment variable.
• lines 775-793, parseProfilerProtocol: unset means http/protobuf, and grpc (any case) selects OtlpGrpcProfilesExporter.
• lines 672-685, resolveProfilerEndpoint: OTEL_AWS_PROFILER_ENDPOINT is used verbatim for both transports.

A gRPC container pointed at the HTTP endpoint would pay the profiler's cost and deliver nothing, so it is skipped. A container that sets its own OTEL_AWS_PROFILER_ENDPOINT is still profiled with that endpoint. Profiling gRPC apps, with a gRPC receiver and per-protocol routing, is planned as a separate follow-up.

Why 160Mi per container

Measured data-dir peaks with ADOT's default settings were 27 MB on a Spring Boot service with 200 request threads, and 70 MB on a synthetic JVM with 100 threads. ADOT caps the rotated files at 128 MiB but never deletes the window being written. The limit is a ceiling, not a reservation. Exceeding it gets the pod evicted, which is why values under 16Mi are refused.

Integration test

A new case in the InstrumentationTest job of operator-integration-test.yml, after the existing language steps:

• Enable Java profiling in the operator.
  • Appends --auto-instrumentation-config with java.profiling (enabled: "true", mode: "cpu", volume_size_limit: "64Mi") to the manager container, as the Helm chart does.
  • Waits for the rollout, then dry-runs a Java pod through admission until the reply carries OTEL_AWS_PROFILER_ENABLED. The pod webhook ignores failures, so readiness alone does not prove injection.
• Test for Java profiling injection. sample-deployment-java-profiling.yaml has two Java containers (app, worker) and a non-Java sidecar. go run ./integration-tests/java-profiling/cmd checks every running pod against java_profiling_expectations.json:
  • each profiled container has exactly ENABLED, ENDPOINT, DATA_DIR=/otel-profiler-data-java/<container> and MODE=cpu, each set once;
  • each mounts the volume read-write;
  • the pod has one emptyDir with sizeLimit 128Mi (2 x 64Mi);
  • the sidecar and init containers get nothing.
• Test for Java profiling opt-out. A Java-instrumented deployment whose container sets OTEL_AWS_PROFILER_ENABLED: "false" must keep only that variable, set once, with no other profiler variable, mount or volume.
• Non-vacuous.
  • Every checked container must still carry the javaagent.
  • An expectations file that checks nothing is rejected.
  • The steps also run in digest mode, so validating an older image fails them.
  • validate_instrumentation_vars.go is untouched.
• Envtest twins (in make test).
  • TestJavaProfilingIntegrationCase renders the same fixtures through the pod mutator, with controls that fail when profiling is off.
  • TestJavaProfilingEndpointFollowsAgentConfig creates AmazonCloudWatchAgent resources and checks the endpoint port for each of these cases:
    • a custom port;
    • both OTLP ends;
    • no receiver;
    • no port;
    • an out-of-range port;
    • no resource.

Testing

All of this was run locally on this PR's commit. The logs are local and not linkable; the PR's CI runs will add the links.

• make test steps and make lint pass (Go 1.26.10):
  • generate, fmt, vet, and envtest 1.33 with KUBEBUILDER_ASSETS;
  • go test over all packages: 31 ok;
  • golangci-lint: 0 issues; checklicense and impi pass.
  • A control run without envtest assets fails, which proves the suite ran against envtest.
  • api-docs (crdoc) could not be fetched; there are no CRD changes.
• Reproducing tests, red first. On the previous revision, the new custom-port, Fargate and annotation-ignored tests fail; on this commit they pass.
• Operator Integration Test, run locally in minikube against an image built from this commit with make container. The workflow's run: steps were run unchanged in golang:1.26.
  • InstrumentationTest passes, including the 3 Java profiling steps (injection, and opt-out through OTEL_AWS_PROFILER_ENABLED=false).
  • Deployment-, Daemonset-, Statefulset- and NamespaceAnnotationsTest pass.
  • The image was built from this commit's code. The final commit differs from it only in comments.
  • Local differences: cluster starts were serialized; a cert-manager readiness wait was added before make deploy; tool binaries were seeded to avoid GitHub rate limits.
• On EKS, with the chart from feat(chart): add experimental Java continuous profiling via the ADOT java agent aws-observability/helm-charts#422 and its new EKS-JavaProfiling module: Kubernetes 1.33, us-west-2, a test account, and one m5.2xlarge node.
  • Images: this commit's operator image, and the newest CloudWatch agent build that contains the profiles pipeline (a main build from before v1.300075.0, which is not published yet). The agent's OTLP endpoint was enabled with agent.config: default:otel.
  • Endpoints: the service name and injected endpoints are correct: traces and Application Signals metrics go to 4316, and profiles go to the default:otel OTLP HTTP endpoint on 4318.
  • Application Signals metrics and profiles (wall) reached CloudWatch for the service.
  • X-Ray traces did not, for an account reason. The test account sends trace segments to CloudWatch Logs (Transaction Search) without the Logs resource policy that allows it, so the agent's PutTraceSegments calls were rejected with 400. Traces did reach the agent on 4316. The traces check needs a run in the CI account, whose link will be added.
  • This PR's Java profiling integration case passes on that cluster, profiled and opt-out. The expectations were adjusted to the chart's defaults: no mode, the chart's max_payload_bytes, and 2 x 160Mi.
• amazon-cloudwatch-agent-test eks e2e: N/A. That suite has no profiles validator yet, and its EKS tests install released operator and agent images, which do not have this feature.

Response to the CloudWatch Agents team review

1. Receiver ownership. Accepted. Profiling uses the agent's OTLP endpoint under the shared OTLP design and opens no receiver of its own. The operator follows the endpoint's configured port (opentelemetry.collect.otlp.http_endpoint of the cloudwatch-agent agent), falling back to cloudwatch-agent.amazon-cloudwatch:4318.
2. Host-network port conflicts. Covered by the OTLP design; no profiling-specific change.
3. EKS add-on. Covered by the OTLP design. test: fix flaky auto-monitor and TestRateLimit unit tests #422's README notes that profiling is Helm-only until the add-on schema accepts java.profiling.
4. Opt-out annotation. The annotation is removed. The opt-out is OTEL_AWS_PROFILER_ENABLED=false on the container. A namespace or workload opt-out is not needed, so no exclude setting is added.
5. Encryption. Done: test: fix flaky auto-monitor and TestRateLimit unit tests #422's README says profiles are sent unencrypted inside the cluster.
6. Experimental status. Done in test: fix flaky auto-monitor and TestRateLimit unit tests #422's README.
7. monitorAllServices scope. Done in test: fix flaky auto-monitor and TestRateLimit unit tests #422's README.
8. Integration test with Application Signals and profiling. Done: test: fix flaky auto-monitor and TestRateLimit unit tests #422's EKS module runs real ADOT and a real agent with the profiles pipeline, and this PR's integration case also ran on that cluster (see Testing). The gap: no released agent image has the pipeline yet, so a pre-release build was used.
9. Agent memory. Measured with 1, 5, 10 and 20 profiled pods on one node (see test: fix flaky auto-monitor and TestRateLimit unit tests #422). The 512Mi limit is unchanged.
10. Payload size. Done in test: fix flaky auto-monitor and TestRateLimit unit tests #422: max_payload_bytes defaults to 20 MiB. ADOT checks the uncompressed payload against a 64 MiB default, but the agent's OTLP HTTP receiver caps the decompressed body at 20 MiB.
11. No cloudwatch-agent agent. Declined for scope: it is chart behavior beyond making ADOT send to the endpoint.

• Lower priority:
  • test: fix flaky auto-monitor and TestRateLimit unit tests #422 adds tests: Application Signals sections unchanged, True, and the operator argument.
  • test: fix flaky auto-monitor and TestRateLimit unit tests #422's README says the keys follow ADOT.
  • The Fargate skip is here, with a test.

Notes for reviewers

• gRPC detection scope. The system property is read only from JAVA_TOOL_OPTIONS, not from the command line, args, JDK_JAVA_OPTIONS or _JAVA_OPTIONS. A container that selects gRPC only there is profiled, and its exports to the HTTP receiver fail.
• Header forwarding. ADOT attaches OTEL_EXPORTER_OTLP_PROFILES_HEADERS or OTEL_EXPORTER_OTLP_HEADERS to profile exports. A container that sends OTLP to a third-party backend with such headers, and has no own OTEL_AWS_PROFILER_ENDPOINT, sends those headers to the in-cluster agent.
• Agent lookup failures. If the AmazonCloudWatchAgent cannot be read at admission, the endpoint falls back to 4318. The existing helper drops the lookup error the same way for the default Instrumentation.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

PR Checklist

• Commits are squashed into a logical, reviewable set (one commit for a single change): one commit
• Commits and PR description comply with Amazon internal guidelines
• make passes locally (build, unit tests, lint): the make test steps and make lint (see Testing)
• All GitHub Actions checks on the PR are passing
  • Pending: CI runs on this revision once it is pushed. Tick this when it is green.
• Integration test evidence: link to a passing run, or state N/A with the reason
  • Local runs on this commit's code:
    • every minikube job of operator-integration-test.yml passes;
    • this PR's Java profiling case passes on EKS with test: fix flaky auto-monitor and TestRateLimit unit tests #422.
  • Logs are local and not linkable; this PR's CI run will add the link.
• New or updated integration test coverage: link to the corresponding amazon-cloudwatch-agent-test PR, or state N/A
  • N/A: the coverage is added in this repo (integration-tests/java-profiling) and in test: fix flaky auto-monitor and TestRateLimit unit tests #422's EKS module. amazon-cloudwatch-agent-test has no profiles validator, and its EKS tests use released images without this feature.
• Related PRs in any repo are linked in this description and link back to this PR, with any required merge or release order stated: feat(chart): add experimental Java continuous profiling via the ADOT java agent aws-observability/helm-charts#422 links back. This PR merges and ships in an operator release first. Profiles need agent v1.300075.0+ (feat(translator): generate a profiles pipeline from the otlp section amazon-cloudwatch-agent#2300).
• New functionality has unit tests; bug fixes have a reproducing test: javaprofiling_test.go, javaprofiling_endpoint_test.go, javaprofiling_integration_test.go, main_test.go, validate_test.go
• Config translation changes include updated golden files: N/A, no agent config translation in this repo
• Breaking or customer-visible changes are called out in the PR description: new opt-in java.profiling block only; nothing changes when it is unset

[miconeilaws]

Thanks for removing the opt-out annotation and the profiling-specific receiver. Both are now in line with the default-on OTLP design. The comments below are what's left before this can merge. Several need matching changes in aws-observability/helm-charts#422, and each comment says which repo it applies to.

Blocking

1. The endpoint has to come from the chart, not from the agent config. (#441, #422)

For now, the Helm chart will manage the OTLP endpoint. The draft aws-observability/helm-charts#425 builds the receiver and its pipelines in the chart's collector YAML (spec.otelConfig, receiver otlp/cw_k8s_otlp) from the new otlp.* values. The agent's JSON config no longer has an opentelemetry block, and #425 refuses to render when the JSON has opentelemetry.collect.otlp and otlp.enabled is true.

This PR reads the port from opentelemetry.collect.otlp.http_endpoint in the JSON. Once #425 merges, the operator never finds that block and always falls back to 4318:

Setting	Endpoint #441 injects	Result
#425 defaults	:4318 (fallback)	Reaches the right receiver only by coincidence
otlp.http.port=14318	:4318	Wrong port; profiles are lost
otlp.http.enabled=false	:4318	Nothing listens there

The chart is what knows the endpoint, so it should pass it to the operator, for example in --auto-instrumentation-config.

• It already has otlp.enabled, otlp.http.enabled and otlp.http.port. Until #425 ships, it has the custom JSON http_endpoint instead.
• The operator then uses the endpoint it is given and stops reading the AmazonCloudWatchAgent resource for profiling. This also keeps the operator from depending on the chart's internal receiver names.
• The chart owns the receiver, so it can pass the matching payload cap with the endpoint (see item 4).

2. Gate injection on an OTLP HTTP endpoint existing. (#441, #422)

If the chart passes no endpoint, inject nothing, and drop the 4318 fallback. This follows the PR's own reasoning for skipping gRPC containers: injecting without a receiver costs the profiler's CPU and delivers nothing.

• The chart passes an endpoint only when otlp.enabled and otlp.http.enabled are both true. Until #425 ships, the condition is a custom JSON http_endpoint.
• Rendering should fail when java.profiling.enabled is "true" and that condition does not hold.

3. data_dir writes outside the profiler volume. (#441, #422)

The volume is always mounted at /otel-profiler-data-java, but data_dir changes OTEL_AWS_PROFILER_DATA_DIR to <data_dir>/<container>. With data_dir: /tmp/prof, for example, profiler data goes to the container root filesystem. That fails with readOnlyRootFilesystem, and the sizeLimit no longer applies. Please remove the data_dir key.

4. Expose only enabled for this iteration. (#441, #422)

Today any key under java.profiling becomes OTEL_AWS_PROFILER_<KEY>. The chart only checks that the values are strings, and the operator injects unknown keys and logs a line. As a result, a typo such as wall_interval_msec renders, deploys and silently does nothing, and the chart's public interface becomes ADOT's env var names, which can change without any change on our side.

Please make the interface a schema, and for this iteration let it contain only enabled. Is there a setting you need beyond that? Our reasoning for the others:

• volume_size_limit: the operator should own it. ADOT caps rotated files at 128 MiB, and the measured peaks were 27 MB and 70 MB, so the fixed 160Mi default covers ADOT's defaults. It only needs to come back if we expose settings that grow the data, such as window_seconds or memory_enabled.
• max_payload_bytes: this works around a mismatch between ADOT's 64 MiB default and the agent receiver's 20 MiB limit. A customer gains nothing by changing it, so the operator should always inject the cap the chart passes with the endpoint.
• data_dir: removed; see item 3.
• ADOT settings (mode, window_seconds and so on): add them one at a time when there is a use case. For each, decide whether it also brings back volume_size_limit.

In the chart, reject any key other than enabled at render time (values.schema.json or a template check), and update the README and values.yaml to match. In the operator, drop the generic OTEL_AWS_PROFILER_<KEY> mapping.

5. Don't skip containers that use envFrom with a Secret or a prefix. (#441)

injectJavaProfiling skips any container with envFrom.secretRef or a prefixed envFrom. Many real workloads load credentials this way, so they are silently not profiled; the only sign is an Info log line in the operator. The existing javaagent and Application Signals injection don't skip these containers: their env entries override envFrom values, and that behavior is accepted. Please follow the same rule for profiling, and document that a container's own OTEL_AWS_PROFILER_ENABLED must be set in env or a ConfigMap to take precedence.

6. CI is failing. (#441)

• "Run Unit Tests" fails in Test_OptOutByDisablingMonitorAllServices/StatefulSet/auto_restart_true (pkg/instrumentation/auto).
• The NamespaceAnnotationsTest job fails in TestAllLanguagesNamespace.
• This PR touches neither package, and "Push on main" is green at 5710ae66. Please re-run them, or show that they fail on main too.