Sprint 1 - AgentOps CLI Baseline #5

Goal

Deliver an installable CLI capable of running a baseline Microsoft Foundry evaluation bundle from a YAML configuration and producing normalized outputs.

CLI Approach

Scope

• CLI skeleton and packaging
• Evaluation config schema validation
• Baseline evaluation bundles (RAG + tool agent)
• Minimal reference dataset
• Normalized results schema
• Simple CI workflow for automated evaluation
• Package and publish AgentOps Toolkit to PyPI

Deliverables

• agentops eval run command
• results.json + summary.md outputs
• Sample evaluation config
• GitHub Actions workflow
• Quickstart documentation
• Published PyPI package (pip install agentops-toolkit)

Definition of Done

• CLI installs and runs locally
• Evaluation executes end-to-end
• Threshold violations fail CI
• Output schema versioned and stable