feat(doctor): change attribution, explain what changed since the last comparable run #494

Summary

Doctor can tell you that a metric regressed, but it gives you no way to see why. Today the only path is to manually dig through git history and guess which prompt or agent change lined up with the drop.

This issue proposes a change-attribution feature: record the git commit behind every eval run, then use that to diff the prompt file and agent target between the latest run and its most recent comparable baseline. The result surfaces in two places:

1. A new deterministic Doctor check (change_attribution.prompt_diff) that reports the change in plain language.
2. A new Cockpit page (/api/version-timeline) showing the same story across the whole run history, newest first.

No LLM in the loop — the whole thing is deterministic, cheap, and unit-testable without a m

Motivation

• A regression finding wiser to leave the tool toinvestigate.
• We already store enough this possible; we justdon't record which commit produced the run.
• Both Doctor and Cockpitnged" means, so they needto share one diff implementation.

Proposed design

1. Record the git SHA on every eval run

New helper agentops/core/git_info.py::resolve_git_sha(workspace=None):

• Checks CI env vars first (GITHUB_SHA, BUILD_SOURCEVERSION,
  Build.SourceVersion) siys correct in a pipeline.
• Falls back to git rev-parse HEAD for local development.
• Never raises: missing or timeout all return ""`.

Every run writer adds two→ config:

Touched writers: pipeline/orchestrator.py (local + cloud snapshots),
pipeline/azd_runner.py,.py.pipeline/prompt_deploy.py::_git_sha()` is refactored to reuse the same helper
(it previously only read

2. Expose the new fie

RunSummary gains git_s, and a newdataset_evaluators_fingerprint.

The new fingerprint hashes (dataset, evaluators) only — deliberately
excluding the agent targethodology_fingerprint`. The regression check needs the target to match; the attribution check needs to
compare runs across an e explaining that bump isthe whole point. It still refuses to compare runs that used a different
dataset or evaluator set, be misleading.

_methodology_fingerprint_dataset_and_evaluators()and _hash_fingerprint_payload() with the new one — no behaviour change.

3. New Doctor check: change_attribution.prompt_diff

agentops/agent/checks/change_attribution.py:

• Takes the latest run and its most recent baseline with the same `dataset_evaluators_finge
• Bails out silently (no finding) when: fewer than min_runs, no compbaseline, either run has ourced runs), or both runsshare the same SHA. - Runs git diff <old_shahs> (best-effort, 10 stimeout, returns None on any failure so a Doctor run can never crashgit is unavailable or his
• Emits one INFO finding under Category.QUALITY with a plain-langusummary, e.g.:
  > Between these runs, t(s) added, 4 removed) andthe agent target changed: my-agent:1.2.0 -> my-agent:1.3.0.
  Evidence carries latest_run_id, baseline_run_id, old_git_sha, new_git_sha, diff_summs to the exact git diffcommand to run. summarize_prompt_diff() is a pure function so it's testable without a real git repo. Config (agentops.agent.cattribution`):
  ```yaml
  checks: change_attribution:
  enabled: true min_runs: 2
  prompt_paths: [] # files to diff, relative to workspace; falls back to the run's own prompt_file

Registered in checks/cataults_history", "workspace") and wired into analyzer.analyze() right after the regression check.

4. New Cockpit page: version timeline

GET /api/version-timeline renders the last 50 runs, newest first. Each row
shows pass/fail badge, ru), timestamp, target, short SHA, and a one-line change summary vs. the run immediately before it:

• "First recorded run."
• "Same commit as the pre
• "No commit recorded for one of these runs - re-run agentops eval run to
  start tracking changes.
• Otherwise the same summarize_prompt_diff() sentence as the Doctor check.

It reuses _git_diff_stat / summarize_prompt_diff from the check so the two
surfaces never disagree. mplate gets a heading slotand a "version timeline" link in the header; _project_run() now also projects
git_sha, prompt_file, and

Scope of the spike

Files touched (≈730 inser

• src/agentops/core/git_i
• src/agentops/agent/checks/change_attribution.py (new) src/agentops/agent/checzer.py, agent/config.py
• src/agentops/agent/sources/results_history.py src/agentops/agent/cock
• src/agentops/pipeline/{orchestrator,azd_runner,azd_eval_runner,prompt_d}.py
• tests/unit/test_agent_checks_change_attribution.py (new), tests/unit/test_cockpit
  t of scope / follow-ups
  LLM-generated explanatif means semantically (thischeck is intentionally deterministic; an llm_assist variant could layertop later).
• Diffing anything beyond the prompt file and agent target (datasets, evaluator configs, thre
• Attribution for cloud-sourced runs that carry no git_sha. Rendering the actual dintly only the line-countsummary).
  Plan
• Split the spike into reviewable commits (git SHA recording → resulthistory → Doctor check
• Docs: new check in the Doctor checks reference, change_attribution config block, Cockpit t
• Decide whether prompt_paths should also accept globs
• Open PR against dev