agent-assay

An evaluation-integrity layer for tool-calling agents: it takes an agent adapter, a resettable environment and a golden set, scores the trajectory, the tool arguments, the final environment state and the outcome, and refuses to report any of it until the harness has been shown it can go red. Negative controls decide admissibility, comparisons across mismatched suites or scorers are refused rather than averaged, and a headline from a single pass is refused unless a control has certified the config as reproducible.

Read LIMITATIONS.md first. Its first item is that the local-model runs below are inadmissible, and why that is a finding about the model rather than a fault in the harness.

The system under test is the banking support agent in ../regulated-support-eval — a sibling checkout, not a dependency you can pip-install. Without it there is nothing here to evaluate.

Every number below is produced by the command written beside it. make readme-check regenerates all of them from the committed artifacts in reports/ and fails on any drift; make number-audit fails if a number in this file or LIMITATIONS.md is neither regenerated that way nor listed in NUMBERS.md with its own command. EVIDENCE.md maps every claim to the artifact and run id behind it.

It reproduces the system under test's own scoring, to the digit

.venv/bin/agent-assay run --config mock --db run/assay.db
.venv/bin/python tools/compare_to_sut.py <run_id> --db run/assay.db

metric	agent-assay	../regulated-support-eval/reports/trajectories.json	match
n	104	104	yes
trajectory_accuracy	0.7980769230769231	0.7980769230769231	yes
mean_lcs_ratio	0.9311844405594406	0.9311844405594406	yes
outcome_accuracy	0.9134615384615384	0.9134615384615384	yes
wrong_tool_calls_total	4	4	yes
missing_tool_calls_total	14	14	yes

Compared with == on floats, not isclose. The aggregate is not the check, though — two compensating per-script disagreements would average out — so every one of the 104 scripts is also compared individually on exact_match, lcs_ratio, actual_outcome and the full actual_path: 0 mismatches. Nothing was tuned to reach it; the first full run produced these numbers.

The controls, and what they measure

.venv/bin/agent-assay controls run --config mock --db run/assay.db   # exit 0

6 PASSED, 1 NOT_RUN, exit 0, run admissible. The bands are measured floors, not zeros — src/agent_assay/controls/expected.yaml carries each one with its derivation.

control	status	what it measured on the mock run
scorer_selftest	PASSED	6 of 6 scorers proved they still go red on a planted defect
shuffled_gold	PASSED	deranged gold: trajectory.exact 0.7981 → 0.0962, state.assertions 0.8846 → 0.0385, outcome.match 0.9135 → 0.7212
state_check_bites	PASSED	flipping one asserted field drops state.assertions 1.0 → 0.0; an unasserted change raises state.collateral 0.0 → 1.0
null_agent	PASSED	0 tool calls across 104 trajectories; exact 0.009615384615384616, outcome 0.038461538461538464, state.assertions 0.023076923076923078; twin margin 0.7884615384615385
random_agent	PASSED	chance over 10 seeds: exact 0.0009615384615384616 (sd 0.0028846153846153848), outcome 0.14326923076923076 (sd 0.013900800283462463), state.assertions 0.023076923076923078 (sd 0.0), tools.recall 0.0767628205128205
identical_config	PASSED	two separate executions: 1519 values compared, max_abs_delta 0.0, 0 differing paths
judge_calibration	NOT_RUN	no judge exists; M6

Two of those numbers are the reason this layer exists rather than a metric library:

• The chance level for trajectory.exact is 0.00096, not the 0.0096 a single random seed suggested. One seed happened to pass the one script a do-nothing agent passes; over 10 seeds it is passed by roughly one seed in ten. A band set from one observation would have been ten times too generous.
• A deranged golden set still scores outcome.match 0.7212 against the mock's 0.9135, because 87 of 104 items expect resolved. That is limitation 2, and it is the reason no headline here is an outcome number.

Mutation: what the scorers would miss

.venv/bin/agent-assay mutate --config mock --operators configs/mutants.yaml \
    --tag card_lost_stolen --db run/assay.db

gate_mutation_score: 1.0 — 5 killed of 5, 0 equivalent, over the 23 items tagged card_lost_stolen. There are no equivalent mutants to list, and that is a computed result: each operator was checked for equivalence against the baseline's trajectories before the run and by counting the proposals it actually altered after it — 45, 23, 90, 45, 54, none of them zero.

The per-metric table is the finding, not the 1.0:

metric	baseline	drop_tool_call	corrupt_arg	swap_call_order	duplicate_call	skip_state
trajectory.exact	0.9130	0.0000	0.9130	0.0000	0.0000	0.0000
tools.recall	0.9783	0.0000	0.4783	0.6957	0.9783	0.0000
state.assertions	1.0000	0.0000	1.0000	0.7500	1.0000	0.0000
args.match	1.0000	0.0000	0.4348	0.6522	1.0000	0.0000
tools.redundant	0.0000	0.0000	0.0000	0.0000	1.6087	0.0000
tools.order_violations	0.0000	0.0000	0.0000	0.0870	0.0000	0.0000
outcome.match	0.9565	0.0870	0.9565	0.7391	0.9565	0.0000

corrupt_arg reads another customer's account and three of the four detector metrics call it correct. On card_lost_stolen-happy-006:

--- baseline
  turn 2 get_account({'account_id': 'SYN-ACCT-0007'}) ok=True
  turn 4 block_card({'card_id': 'SYN-CARD-0007', 'reason': 'reported_lost_or_stolen'}) ok=True
--- corrupt_arg(account_id)
  turn 2 get_account({'account_id': 'SYN-ACCT-0037'}) ok=True
  turn 4 block_card({'card_id': 'SYN-CARD-0007', 'reason': 'reported_lost_or_stolen'}) ok=True

The path is identical, the world ends up correct, the outcome is correct — so trajectory.exact 0.9130, state.assertions 1.0000 and outcome.match 0.9565 are all right. Only args.match (1.0000 → 0.4348) and tools.recall (0.9783 → 0.4783) see it. A harness without argument-level scoring would report this run as perfect, and what it would be hiding is a privacy incident.

The local model: three passes, and an inadmissible verdict

export EVAL_MODEL=openai-compatible:qwen2.5:7b
export EVAL_MODEL_BASE_URL=http://localhost:11434/v1 EVAL_MODEL_API_KEY=ollama
.venv/bin/agent-assay run --config qwen25-7b-t07 --repeats 3 --limit 20 --db run/assay-v2.db
.venv/bin/agent-assay controls run --config qwen25-7b-t07 --limit 20 --db run/assay-v2.db \
    --allow-not-run judge_calibration --allow-not-run identical_config     # exit 2
.venv/bin/agent-assay report 953271a31c5449c2a3a4497cbed572cb --db run/assay-v2.db --headline-only

qwen2.5:7b on local Ollama, temperature 0.7, seed unpinned, over the 20-item adversarial subset. The run is inadmissible — null_agent, random_agent and shuffled_gold all FAIL — and the report says so before it says anything else:

!!! INADMISSIBLE -- the numbers below are not reportable evidence
    reason: failed control(s): null_agent, random_agent, shuffled_gold

The reason is a finding: on exact-path agreement this model is indistinguishable from an agent that proposes nothing. Both score trajectory.exact 0.0; the null twin requires a margin of 0.5 and measures 0.0. On outcome and on final state it is distinguishable from doing nothing, which is why the numbers are shown rather than withheld:

metric	pass^3	pass@3	mean of k	null agent, same 20 items
trajectory.exact	0.0000 [0.0000, 0.1611]	0.0000 [0.0000, 0.1611]	0.000000	0.0
state.assertions	0.2000 [0.0567, 0.5098]	0.4000 [0.1682, 0.6873]	0.333333	0.0
outcome.match	0.4000 [0.2188, 0.6134]	0.6500 [0.4329, 0.8188]	0.533333	0.0

95% Wilson intervals over items; n=20 for the two path/outcome metrics, n=10 for state.assertions (the items in this subset that assert anything). pass^3 0.4000 against pass@3 0.6500 on outcome.match is what repeats are for: a harness reporting only pass@k would put this agent's outcome accuracy at 0.6500, and requiring it to be right all three times drops it to 0.4000.

Per repetition, the spread the pooled mean hides:

rep	trajectory.exact	state.assertions	outcome.match
1	0.000000	0.300000	0.450000
2	0.000000	0.300000	0.600000
3	0.000000	0.400000	0.550000

Consistency (all three paths byte-identical) is 0.6000, [0.3866, 0.7812], n=20 — the same 0.60 the system under test reports for the same model on the same scripts, and only 2 of the 8 unstable scripts are the same ones. See limitation 7.

Temperature 0, twice over the same 20 items: consistency 1.0000 [0.8389, 1.0000], pass^2 equal to pass@2 on every metric, and every score value equal to the last digit except latency.total_ms.

.venv/bin/agent-assay run --config qwen25-7b-t0 --repeats 2 --limit 20 --db run/assay-v2.db

That is a measurement of this endpoint on this machine, not a licence: deterministic() still returns False for the config, and report still refuses its k=1 headline. Limitation 8 says why.

A single pass is refused as a headline

.venv/bin/agent-assay report a425ac9f3ae842729238487aa64466d1 --db run/assay-v2.db \
    --headline-only --no-write

run a425ac9f3ae842729238487aa64466d1  config=qwen25-7b-t0  status=complete
suite_hash: 88fa2bead2971cab8f6a8ab8e528d0d4edf9ab8b1c13865d622d4076734b18ba
!!! single run; not a measurement
    the adapter does not declare this config deterministic; identical_config has not PASSED for this config
    k=1 supports no interval and no consistency figure: with one pass there is nothing for a second pass to agree or disagree with.

sample, n=1 (one run of each item; not a measurement of this agent):

n: 5
trajectory.exact mean: 0.0
trajectory.lcs mean:   0.55
outcome.match mean:    0.4
tools.wrong total:     0
tools.missing total:   5
crashed:               0

It does not hide the numbers. Refusing to show them would push a reader to fetch them from the database, where they would arrive with no heading at all. The refusal is a conjunction of two facts and both are needed: deterministic() is a claim, and identical_config PASSED is the measurement that the claim survived contact with the harness.

A comparison is refused rather than averaged

.venv/bin/agent-assay compare 4ac0c53404ad416cb3b3f380089b328f \
    ee9381a6e6914e1fae60ae64aee3a3b6 --db run/assay-v2.db

COMPARISON REFUSED

  ENV_MISMATCH
    runs disagree on affects_results env fields: 4ac0c53404ad416cb3b3f380089b328f={'adapter': 'regulated_support', 'adapter_version': '1.0.0', 'bank_seed': 20260902, 'sut_git_head': '70d66e6b9966ac44f4ea58d2c3e55280b2644641'}, ee9381a6e6914e1fae60ae64aee3a3b6={'adapter': 'regulated_support', 'adapter_version': '1.1.0', 'bank_seed': 20260902, 'sut_git_head': '70d66e6b9966ac44f4ea58d2c3e55280b2644641'}

Nothing was compared. Re-run with --force to see the numbers anyway; every line will be stamped UNSOUND COMPARISON.

Exit code 2. --force renders the table with every line stamped UNSOUND COMPARISON, and the forced table looks entirely reassuring — every metric identical, every delta zero. That is the point: args.match is absent from it, because adapter 1.0.0 did not record tool arguments, and args.match is the one metric on which the two versions genuinely disagree about what was measured. The comparison reads as sound precisely because the row that would have disagreed is missing.

The gate is fireassay.integrity.assert_comparable, imported rather than reimplemented. SUITE_MISMATCH, SCORER_MISMATCH and RUN_INADMISSIBLE are demonstrated in EVIDENCE.md.

Use it on your own agent

Everything above the adapter works in terms of Item and Trajectory and knows nothing about banking. An adapter is the only part that does:

class AgentAdapter(Protocol):
    name: str
    version: str

    def build_suite(self, source: Path) -> tuple[Item, ...]: ...
    def configs(self) -> Mapping[str, Mapping[str, object]]: ...
    def run_item(self, item: Item, config: Mapping[str, object], *,
                 seed: int, repetition: int) -> Trajectory: ...
    def null_agents(self) -> Mapping[str, Callable[..., object]]: ...
    def deterministic(self, config: Mapping[str, object]) -> bool: ...

    # scoring hooks: facts about your system a generic scorer must not assume
    def arg_normalisers(self) -> Mapping[str, Mapping[str, tuple[str, ...]]]: ...
    def mutating_tools(self) -> tuple[str, ...]: ...
    def order_anchor_tool(self) -> str: ...
    def baseline_snapshot(self, seed: int | None = None) -> dict[str, object]: ...

build_suite must be a pure function of what is on disk — two calls on an unchanged source must give the same suite_hash, which is what every comparison rests on. null_agents is not optional: a suite on which a do-nothing agent scores well is not measuring what it claims to, and running one is the only way to find out.

Then three commands:

agent-assay suite build --db run/assay.db            # golden set -> content-hashed items
agent-assay run --config <name> --repeats 3 --db run/assay.db
agent-assay controls run --config <name> --db run/assay.db   # exit 2 on FAILED, 3 on NOT_RUN

score, report, compare, mutate, mutation score, controls show and migrate are the rest of the surface; agent-assay --help lists them.

Running it

The system under test must be checked out beside this repository as ../regulated-support-eval: the adapter imports it by path, the tests use its Temporal harness, and tools/compare_to_sut.py reads its reports/trajectories.json. A clone without it has nothing to evaluate.

uv venv --python 3.12 .venv
.venv/bin/python -m ensurepip --upgrade
.venv/bin/python -m pip install -e ".[dev]"        # pip, not `uv pip`
.venv/bin/python -m pip install -e ../fireassay
.venv/bin/python -m pip install -r requirements-sut.txt
make verify                                        # ruff, mypy --strict, pytest

make verify runs 375 tests at 96% statement coverage on src/, including the eight that drive a real Temporal dev server. Then, on the mock path, with no Ollama and no network:

.venv/bin/agent-assay suite build --db run/assay.db        # 104 items, one suite_hash
.venv/bin/agent-assay run --config mock --db run/assay.db  # ~1 min; prints the run id
.venv/bin/python tools/compare_to_sut.py <run_id> --db run/assay.db
.venv/bin/agent-assay controls run --config mock --db run/assay.db

The third prints RESULT: reproduced to the digit; the fourth prints the control table above and exits 0. make readme-check and make number-audit need neither a database nor the system under test — they read reports/ only, which is why a reviewer can check the arithmetic before running anything. make artifacts is the other direction: it rebuilds reports/ from a database, so it is the one target that needs both.

Local models are optional and cost nothing: Ollama serving qwen2.5:7b at http://localhost:11434/v1. No paid API is used anywhere in this repository, and cost.usd is recorded as an explicit 0.0 with the evidence local model, price 0 rather than omitted.

What is reused from fireassay

Imported as a path dependency, not copied:

Import	Used for
fireassay.hashing.canonical_json, content_hash	content addressing, under our own aa.item.1 / aa.suite.1 / aa.config.1 prefixes
fireassay.integrity.assert_comparable, ComparabilityReport, ComparisonRefusedError	refusing a comparison across mismatched suite, scorers or environment
fireassay.admissibility.assess, Admissibility	admissible iff zero invariant violations and every control PASSED or allowed NOT_RUN
fireassay.controls.base.ControlOutcome	the control result shape, so our outcomes and fireassay's are one type to assess
fireassay.controls._common.stable_seed, seed_from_suite_hash	deterministic seeding that survives a new interpreter
fireassay.mutation.detector.ThresholdDetector, mutation.score.compute_score, MutantResult, MutationScoreResult	the arithmetic that decides a kill and the mutation score
fireassay.models.Run, Suite, Config, Question, SystemOutput	run/suite/config containers; our Item replaces Question
fireassay.items.core.wilson_ci	every interval in this README
fireassay.score.cost.CostScorer, score.latency.LatencyScorer	cost.usd and latency.total_ms, delegated rather than re-implemented
fireassay.evidence (fireassay evidence --claims tools/evidence_claims.py --check)	make evidence: claims over reports/ with expected values written in code, recomputed and compared, never rewritten

The pattern is copied where the code could not be: our own Control protocol and ControlContext, our own mutation operators, the append-only SQLite triggers, the Scorer protocol with our own ScoringContext, and expected.yaml with our own VALID_CONTROL_KINDS.

Not done

• The LLM judge (M6). judge_calibration is a 0.0.0 stub reporting NOT_RUN on every run. The design — cross-family, brake-only, decomposed questions, sensitivity and specificity against human labels, position and verbosity controls — is fireassay's; none of it is built here.
• A second adapter. One system under test, so "framework-agnostic" is a design claim. The Environment protocol has no concrete implementation at all.
• A whole-suite mutation pass, and a re-measured gate_mutation_score. The stored score is over 23 of 104 items and predates the snapshot-diff migration.
• A whole-suite local-model pass. The Qwen runs cover 20 adversarial items; a 104-item k=3 pass is about three hours on this laptop.
• A statistical mutation detector. The shipped one is a naive threshold and says so in configs/mutants.yaml: no significance test, no baseline variance estimate.
• tools.redundant in the detector metric set. Limitation 9: the mutant it exists to catch is killed by an accident of this system's path notation.
• Full list, with what each one costs a reader: LIMITATIONS.md.

Licence — MIT

See LICENSE.