Skip to content

agentops-pr workflow templates should auto-detect committed baseline file #155

Closed

Description

Surfaced during tutorial validation #132 (closed PR #154 prototyped the change but is scoped out of the doc-only PR series).

Problem

docs/tutorial-baseline-comparison.md Section 4 tells users:

The agentops-pr.yml workflow shipped by agentops workflow generate already supports this — drop a baseline file in your repo (e.g. .agentops/baseline/results.json) and add this step: ...

But the shipped templates run agentops eval run --config <cfg> with no --baseline flag. Users have to manually edit the workflow to consume a committed baseline file. Affected templates:

  • src/agentops/templates/workflows/agentops-pr.yml (GitHub Actions)
  • src/agentops/templates/pipelines/azuredevops/agentops-pr.yml (Azure DevOps Pipelines)
$ grep -c 'baseline' src/agentops/templates/workflows/agentops-pr.yml \
    src/agentops/templates/pipelines/azuredevops/agentops-pr.yml
# both: 0

Proposed change

Make the PR step auto-detect a committed baseline file at .agentops/baseline/results.json and pass --baseline only when it exists. Without the file, behaviour is unchanged.

BASELINE_ARG=""
if [ -f .agentops/baseline/results.json ]; then
  BASELINE_ARG="--baseline .agentops/baseline/results.json"
fi
agentops eval run --config "$AGENTOPS_CONFIG" $BASELINE_ARG

Apply the same guard to both the GitHub Actions step and the Azure DevOps AzureCLI@2 inline script so behaviour is consistent across platforms.

Why it matters

Today the tutorial promises 'just drop the file in the repo', then the user discovers the workflow doesn't honour it. Either the docs need to be softened (and the user has to manually edit the workflow), or the templates should match the doc. This issue is the option to make the templates match.

Prior art

Closed PR #154 implemented this end-to-end (templates + tests + doc). It was closed to keep the validation series doc-only; the work is preserved in the closed PR diff for whoever picks this up.

Activity

  1. added a commit that references this issue on Jun 14, 2026
    e2ad606
  2. placerda commented on Jun 14, 2026

    @placerda
    Contributor

    Shipped via PR #306 and released in v0.4.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions