Skip to content

docs: AgentOps lifecycle diagram and quick-reference card #98

Closed

Description

Summary

Create a visual lifecycle diagram (Mermaid) and a quick-reference card that developers can use as a cheat sheet for AgentOps Toolkit.

Motivation

AgentOps has many moving parts: commands, config files, output files, environment variables, exit codes, and evaluation scenarios. A single visual that shows the full lifecycle helps developers orient themselves. A reference card gives them a 1-page printable summary they can keep next to their keyboard.

Proposed Content

1. Lifecycle Diagram (Mermaid)

Full evaluation lifecycle as a flowchart:

Create Agent/Model
    |
    v
agentops init --> .agentops/ workspace
    |
    v
Configure (run.yaml + bundle + dataset)
    |
    v
agentops eval run --> results.json + report.md
    |                       |
    v                       v
agentops eval compare   agentops report generate
    |
    v
agentops workflow generate --> CI/CD pipelines
    |
    v
PR Gate (exit code 0/2) --> Merge or Block
    |
    v
Tracing (AGENTOPS_OTLP_ENDPOINT) --> AI Toolkit / Jaeger / Azure Monitor
    |
    v
Monitor + Iterate

Should be rendered as a proper Mermaid diagram with decision nodes, color coding by phase (setup, eval, CI/CD, observability).

2. Quick Reference Card

Single-page summary:

Commands:

Command Purpose
agentops init Create workspace
agentops eval run Run evaluation
agentops eval compare Compare two runs
agentops report generate Regenerate report
agentops workflow generate Generate CI/CD workflows
agentops skills install Install Copilot skills

Exit Codes:

Code Meaning
0 Passed — all thresholds met
1 Error — config or runtime failure
2 Failed — thresholds not met

Key Files:

File Purpose
.agentops/run.yaml Run configuration
.agentops/bundles/*.yaml Evaluator bundles
.agentops/datasets/*.yaml Dataset configs
.agentops/data/*.jsonl Dataset rows
results.json Machine-readable output
report.md Human-readable output

Environment Variables:

Variable Purpose
AZURE_AI_FOUNDRY_PROJECT_ENDPOINT Foundry project URL
AGENTOPS_OTLP_ENDPOINT OTLP trace export (e.g. http://localhost:4318)
AGENTOPS_FOUNDRY_MODE cloud or local

Acceptance Criteria

  • Mermaid lifecycle diagram covers setup through monitoring
  • Quick reference card fits on one printed page
  • Both artifacts can be embedded in README or docs
  • Diagram uses color coding or grouping by phase

Context

Activity

  1. placerda commented on May 8, 2026

    @placerda
    ContributorAuthor

    Closing this for now because adoption readiness issues/items will be simplified and tracked differently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions