docs: end-to-end adoption guide (workshop-ready, from zero to production) #95
Copy link
Copy link
Closed
Description
Summary
Create a sequential end-to-end adoption guide that walks a team from zero to production with AgentOps Toolkit. This document serves as the map — each step links to the detailed tutorial. It is also designed to be the base material for a 1-day hands-on workshop.
Motivation
Existing tutorials cover individual scenarios (model-direct, RAG, agent workflow, etc.) but no document covers the full adoption journey. A developer who just installed AgentOps has no single guide that shows the recommended path from first eval to CI/CD gating to production observability.
Proposed Content
Sequential walkthrough:
- Install —
pip install agentops-toolkit - Init workspace —
agentops init, understand.agentops/structure - Configure first eval — choose scenario, create
run.yaml, select bundle + dataset - Run local eval —
agentops eval run, understand exit codes - Interpret results —
results.jsonschema,report.mdstructure, threshold pass/fail - Set up CI/CD —
agentops workflow generate, PR gating, branch strategy - Enable tracing —
AGENTOPS_OTLP_ENDPOINT, AI Toolkit Trace Viewer, Jaeger - Compare runs —
agentops eval compare, baseline strategy, regression detection - Monitor quality — cadence recommendations, golden dataset maintenance, alerting
Each step should be self-contained but link to the detailed tutorial for that topic.
Workshop Mapping
This guide maps to a 1-day (7h) workshop:
| Module | Duration | Guide Section |
|---|---|---|
| Intro and Setup | 30 min | Steps 1-2 |
| First Eval | 45 min | Steps 3-4 |
| Interpret Results | 30 min | Step 5 |
| Scenarios (labs) | 60 min | Step 3 variants (RAG, tools, HTTP, safety) |
| CI/CD | 45 min | Step 6 |
| Tracing | 45 min | Step 7 |
| Compare and Regression | 30 min | Step 8 |
| Continuous Operations | 30 min | Step 9 |
Acceptance Criteria
- Document covers the full journey from install to production monitoring
- Each step links to the relevant detailed tutorial
- Workshop agenda table included with time estimates
- Tested by following the guide from scratch on a clean environment
Context
- Existing tutorials:
tutorial-model-direct.md,tutorial-basic-foundry-agent.md,tutorial-rag.md,tutorial-http-agent.md,tutorial-agent-workflow.md,tutorial-conversational-agent.md - CI/CD guide:
ci-github-actions.md - Tracing guide:
docs/telemetry.md(PR feat: wire OTel telemetry into runner, add telemetry docs #86)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
placerda commentedon May 8, 2026
Closing this for now because adoption readiness issues/items will be simplified and tracked differently.