testing.md

Testing — suites, evals, and golden regression cases

• Tests live under tests/backend/ for backend/service behavior, tests/agents/ for agent-specific checks, and tests/evals/ for evaluator sanity checks plus live-eval contracts.
• Vitest is the root test runner. Keep tests deterministic and independent of installed harness CLIs, API keys, network access, and mutable runtime state unless a test explicitly opts into an integration environment.
• Live trajectory evals check ordered agent behavior such as manager planning, staffing, dispatch, review, QA, and acceptance. Live outcome evals check user-facing results and HITL transitions such as ask_user → awaiting-user → user reply → resume. The deterministic tests in tests/evals/evaluator-sanity.test.js only verify evaluator logic; they are not evidence that an agent performs well.
• When a real failure is found, preserve the smallest useful trace as a JSON fixture under tests/evals/fixtures/ and add an assertion that reproduces it before fixing the implementation. These fixtures form the golden regression set; do not remove one merely because the implementation now passes.
• Golden traces should contain only stable, non-secret event data. Redact prompts, credentials, personal data, and large logs; store the invariant event type, actor, relevant status, and expected result instead.
• Every new regression fixture should include the failure it prevents and be exercised by the live eval runner; evaluator sanity checks should remain in npm test so evaluator changes cannot silently break regression detection.
• Live evals are opt-in and use npm run eval -- --harness <name> --model <id> [--eval manager-trajectory|hitl-outcome|all] [--refresh]. Running npm run eval without selection arguments opens an interactive harness/model/eval selector when a TTY is available; CI and other non-TTY callers must provide explicit flags. Use --help or --list-models to inspect options. The runner asks the selected manager model for a JSON event trace, evaluates it against the golden contract, and writes one result file per test.
• The manager integration eval must handle HITL pauses: when the manager enters awaiting-user, the isolated runner submits a deterministic scope reply through /handleSendMessage so the real resume transition and follow-up harness turn are exercised.
• tests/evals/manager-trajectory.integration.test.js is the deterministic orchestration integration contract: it exercises real HR, task, progress, messaging, and harness-dispatch services in isolated state. It is distinct from the live model trace eval and does not require a harness CLI or network.
• npm run eval:integration -- --harness <name> --model <id> [--effort Low|Medium|High|Extreme] runs the opt-in LLM-driven manager integration trajectory. It starts an isolated backend, invokes the selected manager model, and uses reusable fake collaborators from tests/evals/helpers/fakeAgents.js to advance HR, worker, and QA state between manager turns. Its result is authoritative backend-state evidence, not a model-authored event trace. The per-turn collaboration policy (plan detection, multi-role staffing, implementation-task selection, review/acceptance matching, QA targeting) lives in tests/evals/helpers/integrationDriver.js with regression coverage in tests/evals/manager-integration-driver.test.js.
• The LLM integration runner fails closed if the selected harness falls back to the simulator; simulated output is never accepted as manager behavior.
• The current OpenCode integration failure and follow-up hypotheses are documented in research_docs/opencode-manager-integration-eval-investigation.md; preserve fail-closed behavior while investigating permission-bypass flags and MCP startup diagnostics.
• The isolated manager integration prompt omits the shipped tool-root path so the selected model cannot escape into or inspect the core repository; only the temporary project workspace is presented as its workspace.
• Each live eval invokes the harness from a temporary isolated evaluation workspace, never the core project workspace; the workspace is removed after the scenario and the raw model response is retained in eval-results/. Event extraction accepts a valid JSON trace embedded in a short preamble, suffix, or markdown fence, while the contract evaluator still checks ordered event types and actors.
• Eval prompts ask the model to simulate tool and HITL transitions as JSON events; they must not ask the live user, pause for input, or inspect MCP manifest files.
• eval-results/ is intentionally ignored by git. Result filenames include model, harness, test name, and timestamp; result JSON also records the raw response, parsed events, pass/fail status, timing, whether the harness fell back to simulation, and evaluation failures. Do not place secrets in prompts or result output.