Cloud Trace Data Capture #1179

[JuanCaCoder]

First draft of a pull request for Cloud Trace Data Capture.

[netlify]

✅ Deploy Preview for adk-docs-preview ready!

Name	Link
🔨 Latest commit	a1df97e
🔍 Latest deploy log	https://app.netlify.com/projects/adk-docs-preview/deploys/6a88d6ddac20fb000807ca5c
😎 Deploy Preview	https://deploy-preview-2168--adk-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[JuanCaCoder]

📝 Executive Summary

• Updates the Google Cloud Trace integration guide to document telemetry attribute schemas, deployment privacy defaults, and payload redaction behaviors.
• Clarifies architectural boundaries between the Agent Development Kit orchestration layer (ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS) and the model telemetry layer (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT), detailing empty JSON placeholder redaction ("{}") to prevent accidental PII leakage in production.
• Rendered preview: docs/integrations/cloud-trace.md.

• Preview URL: Google Cloud Trace Observability Guide

---

🗂️ PR Consolidation & Verification Mapping

PR / Source Reference	Original Section	New Section	Change Description	Codebase Verification
PR #1179 (adk-docs)	### Captured attributes (bullet list)	### Captured attributes	Converted bullet list into a structured Markdown table covering standard (gen_ai.*) and custom (gcp.vertex.agent.*) trace attributes.	✅ PASS src/google/adk/telemetry/tracing.py:L174-L523
PR #1179 (adk-docs)	None (new section)	### Data privacy and payload redaction	Documented deployment default disabling span content, redacted payload placeholder mechanics, and opt-in environment variables.	✅ PASS src/google/adk/cli/cli_deploy.py:L1183-L1184 src/google/adk/telemetry/tracing.py:L257-L522
PR #1179 (adk-docs)	None (new section)	!!! note (Admonition)	Added Material for MkDocs note distinguishing agent orchestration payload knobs from underlying OpenTelemetry model logging knobs.	✅ PASS src/google/adk/telemetry/context.py:L151-L220

---

🔍 Technical Verification Report

1. CLI Deployment Default Injection (ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS):

   • Citing: src/google/adk/cli/cli_deploy.py:L1175-L1185
   • Verified: When executing adk deploy agent_engine with --otel_to_cloud / --trace_to_cloud, ADK automatically sets ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS='false' in env_vars if not already specified in the user's environment configuration.

2. Core Span Content Resolution & Truthy/Falsy Evaluation:

   • Citing: src/google/adk/telemetry/context.py:L51-L54, L204-L220
   • Verified: In the core Python runtime, TelemetryConfig.should_add_content_to_legacy_spans evaluates os.getenv(ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS, 'true'). Values in _FALSY_ENV_VALUES = frozenset({'0', 'false'}) disable payload population; all other values enable it.

3. Placeholder Replacement Behavior ("{}") on Redacted Payloads:

   • Citing: src/google/adk/telemetry/tracing.py:L248-L523
   • Verified: When should_add_content_to_legacy_spans resolves to False, all payload attributes are populated with empty JSON string placeholders "{}":
     • gcp.vertex.agent.llm_request: tracing.py:L400
     • gcp.vertex.agent.llm_response: tracing.py:L441
     • gcp.vertex.agent.tool_call_args: tracing.py:L257
     • gcp.vertex.agent.tool_response: tracing.py:L287, L336
     • gcp.vertex.agent.data: tracing.py:L522

4. Telemetry Trace Attributes Registry:

   • Citing: src/google/adk/telemetry/tracing.py
   • Verified exact attribute definitions and instrumentation sites:
     • gen_ai.agent.name: tracing.py:L174 in trace_agent_invocation
     • gen_ai.conversation.id: tracing.py:L175, L540
     • gcp.vertex.agent.invocation_id: tracing.py:L387, L506
     • gcp.vertex.agent.event_id: tracing.py:L280, L322, L392, L508
     • gcp.vertex.agent.llm_request: tracing.py:L248, L339, L396, L400 in trace_call_llm
     • gcp.vertex.agent.llm_response: tracing.py:L249, L341, L438, L441 in trace_call_llm
     • gcp.vertex.agent.tool_call_args: tracing.py:L254, L257, L321 in trace_tool_call
     • gcp.vertex.agent.tool_response: tracing.py:L284, L287, L333, L336 in trace_tool_call & trace_merged_tool_calls
     • gcp.vertex.agent.data: tracing.py:L513, L522 in trace_send_data

5. OpenTelemetry SemConv Content Capturing Resolution:

   • Citing: src/google/adk/telemetry/context.py:L55-L81, L151-L202
   • Verified: ContentCapturingMode supports NO_CONTENT, EVENT_ONLY, SPAN_ONLY, and SPAN_AND_EVENT.
   • Truthy values ('1', 'true') for OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT coerce to EVENT_ONLY (logging to LogRecord), whereas span-level capture requires SPAN_ONLY or SPAN_AND_EVENT together with OTEL_SEMCONV_STABILITY_OPT_IN='gen_ai_latest_experimental'.

6. Automated Unit Test Verification:

   • Citing: tests/unittests/telemetry/test_spans.py:L748-L910
   • Verified: Complete test assertions confirm that setting ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS='false' injects '{}' across test_call_llm_disabling_request_response_content, test_trace_tool_call_disabling_request_response_content, test_trace_merged_tool_disabling_request_response_content, and test_trace_send_data_disabling_request_response_content.

[joefernandez]

src/google/adk/telemetry/tracing.py:L174-L523

@JuanCaCoder
Fix your technical report so all your code references are full URL links that go directly to the code repository.

[joefernandez]

Verified every claim in this PR against google/adk-python @ 01919b3 (main HEAD at time of review). All citations below are permalinks pinned to that commit.

Verdict: substantially correct — one required fix.

Verified correct

• All 9 attributes in the new table exist. The five newly documented payload attributes are written at tracing.py#L305-L309 (tool_call_args), #L335-L339 (tool_response), #L629-L633 (llm_request), #L660-L674 (llm_response), #L749-L758 (data). "Prompt text and configuration" for llm_request is precise — _build_llm_request_for_trace emits {model, config, contents} (#L815-L862).
• Redaction to "{}" — each attribute has a matching else branch: #L310-L311, #L340-L341, #L634-L635, #L675-L676, #L759-L760. The "replaced, not dropped" framing matches upstream prose at telemetry_config/index.md#L225-L227, with regression tests at test_spans.py#L1140-L1183.
• 'false' / '0' semantics — context.py#L107-L112 plus _FALSY_ENV_VALUES at #L54. Defaults on, consistent with the "explicitly set 'true'" framing.
• OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT='true'/'1' → events, not spans — coerced to EVENT_ONLY at context.py#L93-L104, which is "content on the emitted LogRecord only" (#L66). The PR's word choice ("events") is correct.
• --otel_to_cloud injects the false default — cli_deploy.py#L1273-L1282.

Required fix: "(or --trace_to_cloud)" is incorrect

For agent_engine that flag is deprecated and takes no part in this behavior:

• Docstring: "trace_to_cloud (bool): Deprecated. This argument is no longer required or used." — cli_deploy.py#L1024-L1025
• Passing it only emits a DeprecationWarning pointing at --otel_to_cloud — #L1210-L1216
• The redaction block is guarded solely by if otel_to_cloud: — #L1273
• The CLI option itself is labelled deprecated, with a _deprecate_trace_to_cloud callback — cli_tools_click.py#L2574-L2581

This was probably inherited from the stale CLI example higher up in the same file (docs/integrations/cloud-trace.md:122 still shows --trace_to_cloud), which is worth fixing in this PR too.

Worth tightening (non-blocking)

1. "automatically sets" omits the if unset condition — the guard is if 'ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS' not in env_vars: (cli_deploy.py#L1281). A value already in .env wins.
2. The default does not apply when telemetry is enabled via .env alone — GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true flips otel_to_cloud in the else branch, after the redaction block is skipped (#L1283-L1292). That route exports traces with content — a real PII footgun the doc currently implies is covered.
3. The default is agent_engine-only — ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS appears exactly once in src/google/adk/cli/. adk deploy cloud_run and adk deploy gke inject nothing.
4. Missing the per-request override and admin lock — RunConfig.telemetry=TelemetryConfig(capture_message_content=...) takes precedence, and setting it to anything (including EVENT_ONLY) bypasses the env var and turns legacy span content off (context.py#L260-L273). ADK_TELEMETRY_IGNORE_RUN_CONFIG=1 locks the policy against application code (#L193-L200) — arguably the most important control for a production-facing privacy section. Consider linking the canonical guide: docs/guides/telemetry/telemetry_config/index.md.
5. Nit: one span writes "N/A", not "{}" — trace_merged_tool_calls sets gcp.vertex.agent.tool_call_args to "N/A" unconditionally (tracing.py#L528-L531). "Placeholder values" would cover both cases.
6. Nit: gcp.vertex.agent.session_id is set but unlisted — written on call_llm spans alongside gen_ai.conversation.id (tracing.py#L624-L626). Users filtering in Trace Explorer will see it.

Suggested replacement for the first bullet

• Deployment default: When you deploy with adk deploy agent_engine --otel_to_cloud, ADK sets ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS='false' for you, unless your .env already defines it. This default applies only to Agent Engine deployments made with this flag: adk deploy cloud_run, adk deploy gke, and enabling telemetry solely via GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY in .env do not apply it, and will export message content unless you set the variable yourself. The deprecated --trace_to_cloud flag has no effect on content capture — use --otel_to_cloud.

[JuanCaCoder]

Updates added as requested.