test: validate eval traces in Azure Monitor (App Insights + GenAI blade) #88

Objective

Validate that AgentOps eval traces arrive correctly in Azure Monitor when AGENTOPS_OTLP_ENDPOINT points to an Application Insights OTLP ingestion endpoint, and document the experience.

Background

AgentOps emits eval traces via OTLPSpanExporter using three semantic convention layers:

• CICD (cicd.pipeline.*) — eval run as pipeline
• GenAI (gen_ai.*) — agent/model invocation attributes
• AgentOps (agentops.eval.*) — scores, thresholds, pass/fail

These traces go to any OTLP backend (Jaeger works locally). For production on Azure, the natural target is Application Insights via its native OTLP ingestion.

What to validate

1. Traces arrive — set AGENTOPS_OTLP_ENDPOINT to https://<region>.applicationinsights.azure.com and run an evaluation
2. Transaction Search — confirm traces appear in App Insights → Transaction Search with correct span names and hierarchy
3. GenAI blade — check if the Generative AI blade in App Insights recognizes gen_ai.* attributes (token counts, model names, etc.)
4. Custom dimensions — verify that agentops.eval.* attributes are accessible as custom dimensions for KQL queries
5. Authentication — document whether OTLP ingestion requires a connection string header, API key, or managed identity

Acceptance Criteria

• End-to-end eval trace visible in Application Insights
• Document what works and what doesn't (GenAI blade recognition, attribute mapping)
• Document any auth/config required beyond setting the endpoint URL
• Update docs/telemetry.md Azure Monitor section with findings
• If changes to telemetry.py are needed (e.g., auth headers), implement them

Notes

• Foundry portal Tracing tab will NOT show these traces — it only shows agent execution traces from the Agent Framework SDK
• This is about eval observability in App Insights, not agent observability in Foundry