AgentOps CLI Foundation and Repository Setup #4

Objective

Establish a production-ready Python CLI foundation for the Microsoft Foundry AgentOps Toolkit that is installable via pip and structured for long-term extensibility, including support for standardized evaluation scenarios across supported agent runtimes.

Scope

This epic covers repository structure, packaging configuration, CLI entry point creation, and implementation of three evaluation scenarios:

• Model-Only
• RAG
• Agent with Tools

All scenarios must support single-turn and multi-turn execution and run on supported runtimes.

Tasks

• Create project structure using src layout

• Add pyproject.toml with:

  • Project metadata
  • Python version requirement 3.10+
  • Minimal dependencies
  • Console script entry point exposing agentops

• Implement CLI framework using Typer or Click

• Implement base command group:

  • agentops --help
  • agentops --version

• Implement eval command group:

  • agentops eval run

• Implement evaluation scenarios:

  • Model-Only (no retrieval, no tools)
  • RAG
  • Agent with Tools

• Support single-turn and multi-turn datasets

• Implement runtime adapters for:

  • Agent Service v2
  • Agent Framework

• Ensure clean import behavior – no side effects

• Add basic linting configuration

Acceptance Criteria

• pip install -e . exposes agentops

• agentops --help prints structured help

• agentops --version prints version

• agentops eval run executes:

  • Model-Only
  • RAG
  • Agent with Tools

• All scenarios run on:

  • Agent Service v2
  • Agent Framework

• Multi-turn cases execute correctly

• CLI exits cleanly with non-zero exit codes on failure

• No runtime errors on startup

Out of Scope

• Advanced dataset management
• CI workflows