eval-quality

Documentation · Getting started · CLI reference · npm

npx eval-quality --help

eval-quality does three things

1. Compile: validate and normalize an eval contract into a machine-readable artifact.
2. Seal: render the brief for the independent evaluator while hiding the planted bug and scoring answer.
3. Preflight: verify baseline environment readiness and probe reachability before running an evaluator.

Scoring is the next milestone: comparing the evaluator’s completed findings with the hidden bug signature to determine whether the bug was actually caught.

What is the eval spec?

It is the test.

More precisely, it is the evaluator’s instructions for how to expose a failure and what evidence counts as finding it.

It defines:

• the behavior being evaluated;
• the probes the evaluator should perform;
• the evidence it should inspect;
• the negative behavior it must rule out;
• the oracle that determines pass or fail.

For example:

Send malformed input. Confirm the request fails. Inspect the full response body. Confirm the expected error. Verify that no record was created.

The planted bug might be:

The API returns the correct error but still creates the record.

A weak eval checks only the response and misses the bug.

A strong eval checks the response and persistence, so it catches the bug.

Caveman summary

Write the eval. Hide the bug. See if the eval catches it.

Key Concepts

Understanding eval-quality requires three core artifacts:

Concept	What it is	Example
Contract (eval-contract.json)	The test specification defining expected behaviors, oracles (checks), permitted tools, and evidence rules.	"Verify API rejects invalid JWT and creates zero database records."
Probe (probe.json)	A diagnostic request sent to the environment to test baseline state, reachability, or fault injection.	A request sending an expired token to /api/v1/resource.
Observation (observation.json)	The empirical response evidence recorded when a probe is executed against the environment.	{ responseStatus: 401, responseBody: { error: "token_expired" } }

How They Fit Together

┌────────────────────────┐      ┌────────────────────────┐      ┌────────────────────────┐
│     Eval Contract      │      │         Probe          │      │      Observation       │
│  (The Specification)   │ ───► │ (Diagnostic Request)   │ ───► │   (Empirical Result)   │
│  "What should happen"  │      │  "Send malformed JWT"  │      │   "Got 401, 0 records" │
└────────────────────────┘      └────────────────────────┘      └────────────────────────┘

Elaboration

Compile disciplined agent eval contracts, then check whether those contracts can catch known bugs.

An agent can produce an answer that reads as correct and is materially wrong. An eval can make the same mistake.

Weak oracle:

Check malformed input is handled correctly.

An evaluator given that instruction sends one malformed request, sees an error come back, and reports success. The record that should never have been created was created anyway. Nobody looked.

Strong oracle:

Send malformed input. Verify the request fails, inspect the full response body,
confirm the specific error, and confirm no record was created.

A passing eval says little when the contract never asked for the probe that would expose the failure. Testing whether the eval can catch a failure you already know about is the first check worth running.

product spec
  → Behavioral Evaluation Contract
  → known defect or gameability probe
  → independent evaluator
  → per-oracle evidence and a gate decision

What each part provides

eval-quality provides:

• the Behavioral Evaluation Contract schema
• the oracle vocabulary and authoring rules
• the contract compiler
• the environment pre-flight
• Eval Contract strength scoring (next milestone)
• versioned evidence output and PASS / WAIVED / CONCERNS / FAIL governance (next milestone)

The caller provides:

• execution of its chosen agent, harness, or person
• repeated trials
• cost accounting
• the live system and environment-probe implementation
• a sealed run record returned for ingestion

eval-quality executes nothing: it never spawns a process, calls a model, drives a system under test, or invokes a judge. Its pure stages are compile, seal, ingest, pre-flight, score, and emit; compile, seal, and pre-flight ship, and ingest, score, and emit are the next milestone. Pre-flight probes the fixture through the environment-probe port, so a contract that declares a fixture reset needs the caller's probe policy to authorize that operation's method as well as the read methods every other pre-flight leg uses. Engine integration is a later adapter behind a port, not a v0 dependency. See ADR-004.

Who it is for

Teams shipping AI agents, coding skills, review bots, MCP-based assistants, or automated test-generation systems, and teams operating human-on-the-loop or dark-factory delivery.

Use eval-quality when all three are true:

• An agent, skill, or model judgment is involved.
• A plausible-looking output can still be materially wrong.
• Observable evidence or probes can expose the wrong behavior.

Deterministic work does not need it and already has cheaper, stronger evidence from unit, integration, contract, E2E and performance testing.

Behavioral Evaluation Contracts

A Behavioral Evaluation Contract is a versioned specification of the behaviors to probe, the evidence to collect, the negative cases to exercise, and the rules that decide whether the system passes or fails. Eval Contract is the shorthand used from here on. The individual checks inside it are oracles. The contract carries no prescribed action sequence; the evaluator chooses its own path.

The authoring discipline is a small set of rules that survived the experiments: separate the success indicator from the body, read the whole body, probe malformed and negative inputs, verify per record, and cross-check sibling parameters and sibling tools.

A compiler enforces these rules mechanically against the contract artifact, in three classes. Structural errors fail compilation. Coverage gaps score down without blocking. A waived pattern is allowed when it records the named rule, a rationale, a machine-checkable condition, and the approval.

Rubrics compile under the same discipline: an anchored scale, a bounded length, named failure-mode penalties, rubric identifiers unique across the contract and criterion identifiers unique inside their own rubric, every criterion stating a question, and every criterion's evidence pointer resolving against the declared interfaces. Authored rubric text that asks a judge to grade the subject's own stated reasoning fails a closed-vocabulary check over the wording.

How Eval Contract strength scoring works

Do not trust a contract because it looks thorough. Put a known defect behind it, run the evaluator, and check whether the contract's oracles caused the defect to be caught.

Two probe classes go behind a contract, and a strong contract rejects both:

• Defect probes, where the behavior is simply wrong.
• Gameability probes, where the behavior looks compliant while dodging the oracle's intent. A test that raises coverage while asserting nothing is the familiar version of this.

Probes come from qualified historical defects or verified controlled mutations. The corpus separates a visible development set from an immutable sealed set for each scoring version.

Every required oracle check resolves to exactly one state, and the state travels with the result, so "the check reported" is never sufficient on its own: caught, confirmed, missed, passed-clean-control, false-positive, abstained, bypassed, unreached, oracle-error, judge-error, infrastructure-error, or not-applicable.

A required oracle that missed, abstained, errored, or is absent prevents PASS, and a high overall score never overrides it. An infrastructure error or a failed environment pre-flight is not a behavioral result at all; it invalidates the run and is re-executed rather than scored.

Using it

eval-quality is its own repository and package, not a plugin inside another framework.

The library is the primary surface. It exports the contract schema, the oracle vocabulary, the compiler, the pre-flight, and the evidence types. The published typed schema is what lets coding agents author contracts correctly by default, which is how the discipline scales beyond the people who went looking for the tool.

The CLI wraps the same library for callers that cannot import TypeScript: CI jobs, GitHub Actions, PR-review and unit-test bots, other frameworks' skills, and any agent permitted to run a shell command.

What the CLI Commands Do

• compile: Typechecks an authored eval-contract.json. Verifies that all behaviors, oracles, rubrics, and sensitivity witnesses comply with structural and authoring rules.
• seal: Generates a sealed-evaluator-brief.json by stripping secret defect signatures, planted answers, and author commentary. The brief carries only the directions and safety bounds the evaluator needs.
• preflight: Reduces caller-supplied probe observations against the contract to verify environment baseline readiness and probe reachability. Halts early with exit code 3 if the environment is unready.

Running the CLI

Every command runs through npx without installing anything:

npx eval-quality compile --in contract.json --out ./eval-out

npx eval-quality seal --in contract.json --out ./eval-out

npx eval-quality preflight --contract contract.json \
  --probes probes.json --observations observations.json \
  --run-id 2026-08-28-a --out ./eval-out

Every command is non-interactive: no prompt, no terminal check, and no behaviour that differs when stdin is a pipe. Each one is a single call into the library plus artifact serialization.

Input and output. An input flag left out reads stdin, and - names stdin explicitly; at most one input may be -. Without --out the artifact goes to stdout, so a command composes with a pipe. An --out ending in .json is a file path; anything else is a directory, and the artifact is written to <target>/<kind>.json where kind is eval-contract, sealed-evaluator-brief, or preflight-verdict. Diagnostics and errors go to stderr, always, so stdout carries the artifact alone.

Exit codes.

Exit Code	Meaning
0	success, and every verdict other than FAIL or a promoted CONCERNS
1	CONCERNS promoted by --strict
2	FAIL
3	invalid: a pre-flight verdict that did not pass
4	structural failure
5	runtime fault
64	usage error

Codes 1 and 2 report a scored verdict. Scoring ships in a later release, so no command here reaches either yet, and --strict changes no code this binary produces. The flag and the two codes are part of the published contract, so they are documented now and wired now.

--strict is the gate-promotion flag and is accepted on every command. --strict-inputs and --no-strict-inputs are a different switch: they set the compiler's input strictness, which is on by default.

The published JSON Schema. A consumer that does not read TypeScript validates against the twelve generated documents, published at the eval-quality/schemas/* subpath:

import spec from 'eval-quality/schemas/eval-contract.schema.json' with { type: 'json' }

The import attribute is required: ESM on Node 22 and 24 both throw ERR_IMPORT_ATTRIBUTE_MISSING without it. The development corpus ships the same way, at eval-quality/corpus/dev/, so an adopter can read real compiled contracts and one compiled-and-sealed pair without cloning this repository.

Relationship with BMad and TEA

The dependency runs one way: TEA uses eval-quality, and eval-quality knows nothing about TEA.

graph LR
  TEA["TEA<br/>(reference authoring client)"] -- "drafts a contract, then calls" --> EQ["eval-quality<br/>(this package)"]

Loading

TEA is the reference authoring client. It reads BMad planning artifacts, notices eval-relevant work, drafts a contract, and calls this package. It is not co-installed, and eval-quality holds no knowledge of TEA, BMad, or any planning-artifact format.

Any human, bot, CI job, skill, or other framework can author a contract and use eval-quality directly. The discipline still applies, because the compiler judges the artifact rather than trusting whoever produced it.

Evaluator runs remain isolated to prevent builder-context leakage and preserve traceability. Stronger contract oracles produced the measured detection improvement.

Real-World Walkthrough: Testing a bmad-tea Knowledge Harness

1. Author an eval-contract.json declaring required knowledge step files (e.g. playwright-utils-mandate.md).
2. Run eval-quality compile --in contract.json to validate contract structure and discipline rules.
3. Run eval-quality seal --in contract.json --out ./run to generate sealed-evaluator-brief.json.
4. Pass sealed-evaluator-brief.json to bmad-tea to execute the task without seeing answer keys.

Evidence and limitations

Holding the model, the budget, the system, and the defects fixed, and changing only how the Eval Contract was authored, sealed-evaluator detection moved from 0.33 to 1.00 across three naturally occurring defects, three repetitions per arm, 19 scored runs.

Both experiment rounds missed at least one preregistered gate. Round 1 recorded DARK-FACTORY REJECTED; round 2 block 1 recorded CONTRACT-DISCIPLINE NOT SUPPORTED, failing one gate of five on a single unreplicated clean control. The separation comes from two of the three defects, since both arms detected the third in every repetition, and both separating cases carry a recorded measurement-layer confound. The sample covers three defects, one system, and one model. This supports a product-direction decision at narrow scale. Certification would require broader replication.

Read the product brief for the product rationale and the PRD for build requirements. The experiment record includes the round 1 verdict, round 2 results, metric summary, and protocol.

Architecture status

The architecture spine is split by pipeline half: the compile-and-seal half is epic-ready, while the score half is not. Gate C closed at zero blocking authoring points and 14 of 14 declaration-only predicates. Gate D's generated-current-fields arm matched the hand-written positive control at 3 of 3 seeded-defect catches, so seal joins the stage-one order without adding an evidence-precondition field.

Contract strength scoring has been open since ADR-007: three rounds of external review established that the catch rate was 1.00 by construction, because nothing matched a finding to the defect its probe seeded. That input now exists and the mapping that reads it is owed to a reference implementation.

Contract compilation was declared ready in ADR-007 and a fourth review withdrew that claim in ADR-008. The named calibration is now complete. The absent local-only mut2 arm was reconstructed from its recorded base, reproduced its prior black-box behavior, and ran under a pre-registered three-arm, three-repetition design. All three arms composed filters and detected the seeded defect in every valid repetition. This closes the calibration gate narrowly; it does not generalize the historical 0.33-to-1.00 effect beyond one behavior and one controlled mutation.

Both are documented as defects rather than dressed as decisions, because four rounds have shown that a confidently worded revision is the thing that goes wrong here.

The decision record, in order: ADR-001 on evaluator isolation, ADR-002 on why authoring discipline is the product, ADR-003 on measurement mechanics, ADR-004 on why this package executes nothing, ADR-005 and ADR-006 on what review and hand-authoring corrected, ADR-007 on the split, ADR-008 on why the other half stopped claiming to be finished too, and ADR-009 on the seventeen places where two conforming implementations still disagreed. Review triage lives in reviews/.

Not building now

Deferred until the contract layer is in real use: claim-to-evidence lineage, semantic checkpoint scoring, process and outcome separation, and first material error attribution.

Out of scope entirely: a new eval engine, a hosted service, a dashboard or GUI, multimodal evaluators, automatic prompt repair, and a generic judge-calibration platform.

Development

npm install
npm run validate            # typecheck, lint, docs, shareable, spine, vectors, schemas, registries, AD-31 table, layers, lineage, boundary, corpus, tests with coverage
npm run build               # emit to dist/
npm run lint:fix            # auto-fix with Biome
npm run test:coverage       # run the suite and fail below AD-30's 90 percent statement and branch floor on core/
npm run generate:schemas    # rebuild schemas/*.schema.json from the Zod source
npm run check:schemas       # fail if the committed schemas differ from the source by one byte
npm run check:ad5-registry  # fail if the failure-code list drifts from the AD-5 table
npm run check:lineage       # fail if a module outside the stage table writes an artifact's lineage fields
npm run check:boundary      # fail if anything the tarball carries references the planning system that produced it
npm run generate:ad31-table # rebuild docs/ad31-coverage-predicates.generated.md from the predicates
npm run check:ad31-table    # fail if the committed AD-31 table differs from the builder by one byte
npm run generate:dev-corpus # rebuild corpus/dev/ from the contract fixtures through the shipped compile and seal
npm run check:corpus        # fail if the committed corpus differs from the builder by one byte
npm run build:shareable     # render the planning artifacts to self-contained HTML
npm run test:conformance    # run the published port conformance suite against every shipped adapter

schemas/ holds the twelve published JSON Schema documents, generated from the Zod definitions and committed. They are the contract for consumers who do not read TypeScript, so they are proven equivalent to the source rather than assumed to be: a byte-exact drift check, a rejection suite asserting the validator keyword and instance path for every negative fixture, a differential check comparing Zod's verdict against a third-party validator's over a generated corpus, and a keyword-mutation sweep that deletes each published constraint and requires some fixture to notice. Edit the Zod schema and regenerate; never hand-edit a file under schemas/.

Every artifact the library hands back is deep-frozen, so it cannot be changed in place. This package is ES modules, which are always strict, so an attempt throws a TypeError there; a sloppy-mode caller sees the write fail silently. A revision is minted as a new artifact carrying its parent's digest and a revision count one greater. check:lineage fails the build when a lineage field is written outside src/core/schemas/, src/core/lineage/, and the modules the AD-24 stage table names as that artifact's producer, which today are src/core/seal/seal.ts and src/core/preflight/reduce.ts.

The eval-quality/conformance subpath publishes the port boundary: the four port types, the message shapes they carry, and an executable conformance suite. An adapter is conforming when runCorpusPortConformance, runClockPortConformance, runFileSystemPortConformance, or runEnvironmentProbePortConformance returns a report whose passed is true, which is the definition rather than a paraphrase of one; each returns a report instead of asserting, so the suite carries no test framework and runs under whichever one you already use.

import { runCorpusPortConformance, type CorpusPort } from 'eval-quality/conformance'

The suite drives a subject through four scenarios and checks six assertions per port method: a mechanism failure is a typed fault, exactly one underlying call happens on success and on failure, an aborted signal rejects promptly, an in-band error value is thrown rather than returned, and a successful call returns a response the published schema accepts. The environment-probe port adds thirteen more from AD-35's default-deny target policy. npm run test:conformance runs the suite against the three adapters this package ships and against an in-repository probe subject that exists only as the suite's own subject.

docs/ad31-coverage-predicates.generated.md holds AD-31's published predicate table, emitted from the seven relevance predicates and their seven satisfaction twins run over a hand-authored contract corpus. It is generated by npm run generate:ad31-table and guarded by npm run check:ad31-table, a byte-exact drift check that fails when a predicate changes and the committed document does not, so the table is evidence the predicates produce rather than documentation kept beside them. Regenerate rather than hand-edit it.

build:shareable renders this README, the product brief, the PRD, the architecture spine, all nine ADRs, and every document those pages link to (contributing, code of conduct, security, licence, and the four experiment records) to _bmad-output/shareable/ as standalone styled HTML for sharing outside the repo. Rendering the linked documents is what lets a recipient without repository access follow the evidence, contribution, security, and licence links instead of hitting a 404; anything that has no page of its own, such as a directory, is marked in the export as needing repository access. Regenerate rather than hand-edit those files: check:shareable fails the build when the committed export is stale or carries a repository URL that is not the canonical one. Mermaid diagrams render as code blocks there, which is a known limitation.

Contributing

See CONTRIBUTING.md and our Code of Conduct.

Security

See SECURITY.md. Please do not open a public issue for vulnerabilities.

License

Apache-2.0 © Murat Ozcan. See LICENSE.