Sprint 3 – Tracing & Telemetry #14

Goal

Bridge evaluation results with telemetry and operational signals.

Scope

• Evaluation-to-telemetry mapping model
• Structured logging format for evaluation signals
• Correlation ID strategy (run_id, operation_id)
• Sample KQL queries
• Skills build/sync housekeeping

Deliverables

• Telemetry convention guide
• Example log emission pattern (runner.py wiring)
• KQL query library

Sub-issues

Telemetry Core (done)

• Add OTLP tracing foundation for evaluation runs #55 — Add OTLP tracing foundation for evaluation runs
• Improve OTLP telemetry span names with dataset context #58 — Improve OTLP telemetry span names with dataset context
• feat: wire OTel telemetry spans into evaluation runner #90 — Wire OTel telemetry spans into evaluation runner
• chore: remove dead agentops trace init CLI command #91 — Remove dead agentops trace init CLI command
• docs: rewrite agentops-trace skill for built-in tracing #92 — Rewrite agentops-trace skill for built-in tracing
• docs: create comprehensive telemetry guide (telemetry.md) #93 — Create comprehensive telemetry guide (telemetry.md)

Telemetry Validation (remaining)

• test: validate eval traces in Azure Monitor (App Insights + GenAI blade) #88 — Validate eval traces in Azure Monitor (App Insights + GenAI blade)
• doc: add KQL query library section to telemetry.md #89 — Add KQL query library section to telemetry.md (depends on test: validate eval traces in Azure Monitor (App Insights + GenAI blade) #88)

Housekeeping

• chore: single source of truth for skills (eliminate drift between templates and plugins) #87 — Single source of truth for skills (eliminate drift)

Definition of Done

• Evaluation signals emitted as OTel spans with structured semconv
• Evaluation signals queryable in Azure Monitor
• Documentation explains trace correlation pattern
• Clear telemetry naming standards established
• KQL query library available in docs