GitHub Actions workflow generation #20

Objective

Provide a ready-to-use GitHub Actions workflow template that runs agentops eval run in CI and produces artifacts, enabling teams to adopt AgentOps evaluation as part of their standard PR/CI pipeline.

Scope

This card covers creating and documenting a GitHub Actions workflow template (and optional helper assets) that developers can copy into their repositories to run AgentOps evaluations in CI with consistent outputs and meaningful exit codes.

Tasks

• Create a workflow template under a predictable path (for example: .github/workflows/agentops-eval.yml)

• Configure workflow triggers:

  • pull_request
  • workflow_dispatch

• Set up Python environment:

  • Python 3.11
  • pip cache enabled

• Install package:

  • pip install agentops-toolkit (or pip install -r requirements.txt + pip install agentops-toolkit, depending on your recommended pattern)

• Run evaluation:

  • agentops eval run --config .agentops/run.yaml (or default behavior if you prefer)

• Upload evaluation artifacts:

  • results.json
  • summary.md (or report.md, depending on Sprint 1 naming)
  • optionally the full .agentops/results/<run>/ folder

• Add documentation snippet explaining how to enable CI in a consumer repo:

  • required files (.agentops/run.yaml, dataset file)
  • required environment variables for authentication

• Ensure CI behavior matches exit code semantics:

  • 0 pass
  • 2 threshold failures (CI should fail)
  • 1 runtime/config error (CI should fail)

Acceptance Criteria

• A workflow template exists in the repo and can be copied to a consumer repository with minimal changes
• Workflow successfully runs agentops eval run in GitHub Actions using Python 3.11
• Workflow fails the job when the CLI exits with 1 or 2
• Workflow uploads evaluation outputs as artifacts and they are visible in the Actions run
• Documentation includes a copy/paste section with the minimal steps to enable CI

Out of Scope

• Implementing new evaluation logic or new evaluators
• Improving threshold gating behavior beyond existing CLI exit code semantics
• Secrets provisioning automation (users manage credentials/env vars)