Skip to content

testing-and-benchmarks.md

Latest commit

 

History

History
141 lines (100 loc) · 3.79 KB

testing-and-benchmarks.md

File metadata and controls

141 lines (100 loc) · 3.79 KB
Raw

Testing and benchmarks

Repository uses language-specific tests plus cross-package verification. Run smallest relevant command first, then broader gates before release.

Core test commands

Go

go test ./...
go vet ./...
go build ./...

This covers Engine, proxy, memory, browser bridge, cache planner, rewriter, and other Go packages in module.

TypeScript and JavaScript

pnpm install --frozen-lockfile
pnpm -r build
pnpm -r test

Individual packages expose narrower scripts:

pnpm --dir packages/cli test
pnpm --dir packages/agent test
npm --prefix extension test
pnpm --dir packages/graders test
pnpm --dir packages/shared/contracts test

Use package manifest to confirm exact scripts.

Python

python -m pytest -q packages/sdk/python

Repository verification

python tests/verify_repo.py

Verifier checks repository invariants that ordinary unit tests do not cover, including generated or packaging expectations.

Platform coverage

Continuous integration builds and tests native Go and JavaScript paths and includes Windows coverage for platform-sensitive launcher behavior. Native release workflow builds a 36-artifact operating-system, architecture, and binary matrix, then publishes signed checksums.

A local pass on one platform does not prove full release matrix. Report exact commands and environment tested.

Benchmark categories

Engine fixtures

Engine benchmarks compare original and compact representations on committed fixtures. Valid report includes selected compressor, recovery result, byte or token counter, and any invariant checks.

Browser fixtures

Browser benchmark measures accessibility-tree capture, query focus, compression, and recovery on recorded pages. See browse/BENCHMARK.md.

Cache corpus

Cache planner corpus tests stress capability and fail-safe gates. Report unsupported and rejected cases; raw pass rate is not a market-ranking result.

Subagent tax

packages/subagent-tax runs local fixtures without provider requests. Its counterfactual results apply to exact fixture, assembly method and token counter.

Wrap report

Published wrap benchmark documents one recorded comparison and provenance. See WRAP-BENCHMARK.md for reproduction availability and claim limits.

Writing a benchmark report

Every public result should include:

  1. question being tested;
  2. fixture names, source, and hashes;
  3. code revision and date;
  4. hardware or provider/model where relevant;
  5. exact command;
  6. count basis and price source;
  7. baseline and treatment definitions;
  8. recovery, parse, and quality checks;
  9. failures, exclusions, and negative deltas;
  10. narrow conclusion supported by data.

A fixture-level result is not production validation. Local token estimate is not provider cost, and successful retrieval does not establish quality parity.

Failure testing

Compression and integration tests should cover:

  • malformed input;
  • duplicate or ambiguous JSON;
  • empty and boundary-sized payloads;
  • transformation larger than original;
  • full or failed recovery store;
  • unknown mode, provider, model, and grader;
  • stream interruption;
  • unsafe endpoint and redirect;
  • secret-bearing headers;
  • unsupported platform or upstream version.

Expected safe result is often original input or explicit rejection. Tests should assert that behavior, not only successful compression.

Documentation verification

Documentation changes need:

  • relative-link check;
  • command and path check against current help and package tree;
  • private or internal reference scan;
  • stale claim scan against code and committed evidence;
  • plain-language review;
  • git diff --check.

Passing documentation checks proves consistency with checkout, not runtime or release readiness.