docs: document AI Toolkit integration with AgentOps Toolkit #94

Objective

Enable developers to export AgentOps evaluation traces and visualize them in AI Toolkit’s Trace Viewer using OpenTelemetry (OTLP).

Scope

AgentOps configuration and validation of end-to-end trace visualization in AI Toolkit. Focus is on ensuring the integration works reliably in a local development environment.

Tasks

• Document how to enable tracing in AgentOps:

  • Configure AGENTOPS_OTLP_ENDPOINT
  • Use HTTP endpoint: http://localhost:4318/v1/traces

• Provide end-to-end integration walkthrough:

  • Set AGENTOPS_OTLP_ENDPOINT
  • Run agentops eval run
  • Open AI Toolkit Trace Viewer
  • Ensure local collector is running
  • Validate traces appear

• Describe expected trace structure:

  • Root span (eval_run)
  • Child spans (eval_item)
  • Evaluator spans (e.g., similarity, coherence)

• Validate compatibility:

  • Confirm spans are correctly rendered in AI Toolkit
  • Ensure timing, hierarchy, and metadata are preserved

• Provide validation guidance:

  • What a successful trace looks like in AI Toolkit
  • How to confirm traces are being emitted

• Document common pitfalls:

  • Missing /v1/traces in endpoint
  • Collector not started
  • Port/protocol mismatch (4317 vs 4318)
  • No traces visible after execution

• Clarify positioning:

  • AI Toolkit is the reference local viewer for this scenario
  • Integration is based on standard OTLP export

Acceptance Criteria

• A developer can run an AgentOps evaluation and see traces in AI Toolkit in under 10 minutes
• End-to-end flow (AgentOps → OTLP → AI Toolkit) works without hidden steps
• Trace structure is correctly visualized in AI Toolkit
• Common issues are documented and easy to troubleshoot

Out of Scope

• Support for other trace viewers
• Deep observability architecture
• Production monitoring setup (Azure Monitor, etc.)
• Custom trace schema design