fix(test-execution): preserve runner failures through CLI and MCP

[rudycelekli]

Summary

aqe test execute currently exits 0 and reports a passing run when one suite passes and another fails to import. Failed teardown and unhandled rejections produce the same false pass; the default test_execute_parallel MCP tool reports 100% passed for these runs even though Vitest exits 1.

Preserve process and suite failures through both execution paths using a shared execution-error check. Return these as execution errors rather than fabricated assertion results, and make CLI execution errors exit 1. Ordinary assertion failures, nested suites, skips and todos retain their test-result behavior.

The explicit compatibility decision for test_execute_parallel / execute-tests is to require a nonempty list of concrete file paths. This handler already rejected wildcard characters before spawning a runner; its old success receipt was a warning with zero executed tests. The public parameter description, types, task documentation and generated examples now state that callers must expand globs before submission. Invalid lists fail as a whole. CLI target discovery remains available through aqe test execute <target>.

Verification

• Final published head db2bd2d2: all 25 checks pass. Full coverage job: 23,624 tests passed / 62 skipped; 977 files passed / six skipped. Coverage artifacts upload successfully; actual line coverage is 65.48%, below the workflow’s existing advisory 80% comparison. No reporter errors or exit normalization occur.

• Original 11-case real-runner regression file on unpatched source: 8 fail / 3 pass; patched: 11 pass.

• Focused regression run: 32 pass across 5 files, including real Vitest service/MCP-handler cases, format controls and CLI exit/evidence coverage.

• Existing coordinator, plugin and handler consumers: 140 pass across 4 files.

• Updated executor task-boundary and real-runner regressions: 55 pass across 2 files (42 executor + 13 real-runner cases). The executor regression verifies that previously rejected file arguments emit TaskFailed, without success data or a TaskCompleted event.

• Actual built CLI and MCP, five scenarios before/after with a fresh MCP process per case: healthy runs remain passing, ordinary assertion failures remain failing, and import/teardown/unhandled errors change from a CLI exit 0 / MCP 100% pass to CLI exit 1 / MCP execution errors.

• npm run build and tsc --noEmit pass. Product lint introduces no findings; one existing unused catch binding remains in the executor (also present on unpatched source).

• Latest concrete-path follow-up: npm run build passed; affected consumer tests 13 passed / 85 filtered; init document generation/preservation tests 3 passed / 31 filtered.

• Local full coverage attempt on Node 22.19 stopped at 3 failed / 6,589 passed / 15 skipped: two knowledge-graph assertions also fail on unpatched source in this environment, and an accessibility timeout passed in isolation on both revisions. This is not a complete passing suite.

• Full GitHub coverage on 48c6494f completed with 2 failed / 23,604 passed / 62 skipped. All 13 real-runner regressions and all 42 task-executor tests pass. The only failures are two timing limits in the unchanged tests/benchmarks/rvf-pattern-store.test.ts (10.157s versus a 10s ingest bound, and a 10s timeout). The benchmark and its native storage dependencies are byte-identical to the original base. Initial uninstrumented controls pass all five cases on both base and head; a later uninstrumented control reproduces both timing failures (10.369s ingest and a 10s test timeout). The benchmark is timing-sensitive even without coverage, so moving it to a different CI job alone is not an established fix. That historical head failed coverage; the current required performance and coverage jobs both pass as reported above.

• Includes the fork-comment guard from fix(ci): preserve reports without unauthorized fork PR comments #701. Report artifacts remain uploaded, while optional comments are skipped for read-only fork tokens; all 12 guard regressions pass.

• Moves the three native RVF timing tests into the mandatory, uninstrumented Performance Gates job. The native availability check and self-search correctness remain under coverage. All original timing assertions, the 1,000-vector load, and 10-second test timeout are preserved. This corrects the measurement lane; it does not optimize native ingestion. The latest local uninstrumented run still fails two original limits (11.94s ingestion and a 10s timeout); the required Linux performance run on db2bd2d2 passes all four native cases (cold start 206.92ms, 1,000-vector ingest 1,506.66ms, search p95 0.50ms). This does not imply host-independent timing. The final full coverage job also passes as reported above.

• Includes the shared runner/reporting correction from fix(ci): execute coherence tests and preserve runner failures #704: nonzero exits remain failures, coverage uses valid reporters, and JUnit stays a test report. Nine wrapper controls and 12 fork guards pass; a real 18-test coverage control produces JUnit, HTML, JSON and JSON-summary with exit 0. An invalid-reporter control runs the same 18 passing tests but retains exit 1.

AQE_MEMORY_BACKEND=memory npx vitest run \
  tests/integration/test-runner-outcomes.test.ts \
  tests/unit/domains/test-execution/test-runner-verdict.test.ts \
  tests/unit/cli/test-command.test.ts \
  tests/unit/domains/test-execution/test-executor-node-test.test.ts \
  tests/unit/domains/test-execution/test-executor-command.test.ts --bail=0

AQE_MEMORY_BACKEND=memory npx vitest run \
  tests/unit/domains/test-execution/coordinator.test.ts \
  tests/unit/domains/test-execution/plugin.test.ts \
  tests/unit/coordination/handlers/coverage-collection.test.ts \
  tests/unit/coordination/handlers/temp-path-rewrite.test.ts --bail=0

Failure modes

• Import failure, healthy/broken suite mix, teardown failure, unhandled rejection, empty suite, and empty/rejected file arguments: covered by the real-runner integration file.
• Nonzero/signal exits, contradictory JSON, text fallback, ordinary Jest assertion messages and Mocha receipts: covered by format tests.
• Nested Vitest suites, intentional skip/todo, healthy runs and ordinary assertion failures: compatibility controls prevent misclassifying valid receipts.
• CLI execution errors now exit 1 without publishing fresh passing evidence; covered by the CLI test and public-interface reproduction.
• CI runner/reporting failures being mistaken for passing tests: covered by tests/unit/scripts/ci-vitest-run.test.ts and the actual valid/invalid reporter controls. Fork comment permissions are covered by tests/unit/scripts/fork-pr-comments.test.ts. Native performance limits remain exercised by the unchanged assertions in tests/performance/rvf-pattern-store.test.ts.
• Rejected, unexecuted requests propagate failure through the task executor; covered by tests/unit/coordination/task-executor.test.ts. Real-runner cases cover a glob-only request and a mixed concrete/glob request, checking that no test-side filesystem marker is written. Existing wildcard rejection is preserved; this change does not add glob expansion.

This closes a remaining failure class after #577 and #592. The runner's JSON can omit an unhandled error's original details; the error reports the available diagnostics and unsuccessful exit rather than inventing an assertion failure.

---

• Every failure mode mentioned in this PR description has either (a) a test that exercises it, or (b) a linked tracking issue.

• Affects published API or CLI surface: yes, execution errors now propagate to CLI/MCP.

• Touches the init flow: generated CLAUDE.md usage example only; its generation, append and existing-section preservation tests pass. No setup logic or publish workflow changes. The full init-corpus gate was not run for this documentation-only example correction.

Additional executor-boundary validation (without AQE_MEMORY_BACKEND=memory, because this existing suite also tests writing result files to its temporary directories):

npx vitest run tests/unit/coordination/task-executor.test.ts tests/integration/test-runner-outcomes.test.ts --bail=0

[proffesor-for-testing]

Adversarial review PASS. Runner/suite failures are separated from assertion failures across the service, CLI, and MCP handler, including import, teardown, unhandled rejection, empty-suite, skip, and todo paths. Local verification: 172/172 focused and affected-consumer tests passed; product CI gates are clean apart from the still-running coverage/dashboard aggregate.

[proffesor-for-testing]

Coverage found a blocking regression outside the focused set: tests/unit/coordination/task-executor.test.ts:468 now receives success: false for the existing testFiles: ["tests/unit/*.test.ts"] contract (23,554 tests passed; this was the sole failure). The new handler turns rejected glob input from a successful warning result into TestRunnerExecutionError, so this needs an explicit compatibility decision and a durable test: either preserve supported glob behavior safely, or update the public/task contract and affected test to require concrete paths. Please patch and rerun the full coverage job before merge. The dashboard permission failure is unrelated.

[proffesor-for-testing]

Re-reviewed after the glob-contract fix. On main, glob paths were already rejected by the safe-path regex and reported as a zero-test success, so turning that into an explicit TestRunnerExecutionError (documented in the MCP schema, types, docs, and init template) is the honest resolution, and the task-executor test now asserts the failure path. Verified the verdict helper is wired through vitest/jest/mocha parsers and both text fallbacks, the coordinator and CLI propagate the error (exit 1, no evidence written), and the MCP task handler returns success:false. Build passes; focused suites pass (task-executor 42/42, real-runner outcomes 13/13 under normal load). Codex adversarial pass flagged two classification gaps that are pre-existing on main and still yield a red result (see comment) — tracked as follow-ups, not blockers.

[proffesor-for-testing]

Thank you, @rudycelekli! A broken test run (suite that fails to import, crashing teardown, unhandled rejection, empty suite) no longer comes back as a fabricated pass from aqe test execute or the test_execute_parallel MCP tool, and the real-Vitest integration tests prove it on both paths. The glob-path decision is the right one: those were already silently rejected, they just looked like a zero-test success. Merged.

Two follow-ups the Codex adversarial pass surfaced, both pre-existing on main and both still producing a red result, just with the wrong classification:

• In executeTestFiles, timeouts, spawn errors, and unparseable output are still converted into synthetic per-file assertion failures rather than a TestRunnerExecutionError, so the CLI reports "N failed" where the MCP handler reports an execution error.
• Mocha counts hook (after/afterEach) crashes in stats.failures, so a teardown crash is classified as an ordinary assertion failure rather than a runner failure.

If you feel like picking those up in a small follow-up PR, that would be very welcome.