fix(quality): enforce measured evidence on the registered MCP gate

[rudycelekli]

Summary

The registered MCP quality_assess({ runGate: true }) tool ignores the flag and decides approval from static source analysis. In a fresh project with a real recorded failing test, it returned passed: true, score 94 and an approve risk decision while the CLI gate rejected the evidence. Complete controlled records also reproduced approval when only tests, coverage or security failed their thresholds.

Route explicit MCP gates through the canonical evidence loader and evaluator already used by aqe quality --gate. Return the seven measured checks and derive the risk decision from their verdict. Missing, incomplete, stale or malformed evidence returns an error. Explicit gates bypass the session response cache so a previous approval cannot survive changed evidence. Reject non-boolean runGate values instead of silently taking the analysis path.

Analysis-only requests retain their numeric static score. Measured gates expose checks and omit qualityScore, because the shared evaluator defines per-metric thresholds rather than an aggregate score; saved reports render that distinction. Public parameter documentation and affected executor tests are updated.

Dependency: this branch includes the exact native graph/obstruction contract fix from #703 (20542b7c). The root integration shard on cc465c20 failed in the unseeded real-WASM fixture with unexpected fallback, matching the failure class fixed by #703 (one failed, 568 passed, 12 skipped). The exact CI random vectors were not logged; #703’s seed-67 control reproduces the connected-graph failure deterministically. The dependency fixes serialization/obstruction handling and uses deterministic connected fixtures; it preserves the fallback and timing assertions. Review and merge #703 first so this PR’s remaining diff focuses on the measured MCP gate.

This addresses the remaining registered-route gap in #596: the separate qe/quality/evaluate implementation already uses measured evidence, but the tool bridge skips it in favor of quality_assess.

CI follow-up

Includes the fork-comment guard from #701 and the runner/reporting correction from #704. Optional fork comments are skipped while artifacts remain available; nonzero runner exits and timeouts remain failures. Coverage uses valid reporters and produces a JSON summary, while JUnit remains a test report.

The 12 fork-comment regressions and nine runner-exit regressions pass. A real 18-test coverage control produces the expected reports with exit 0; an invalid-reporter control retains exit 1 despite all 18 tests passing. Earlier green coverage jobs did not establish completed coverage reporting: the old wrapper could mask the invalid coverage reporter's error. The current full run has now completed successfully with the corrected wrapper.

The MCP Security install on 80bbb986 then stopped before build/tests when optional ONNX CUDA/TensorRT provider downloads from NuGet timed out. The CPU-only MCP workflow now sets ONNXRUNTIME_NODE_INSTALL=skip, supported by the pinned installer. The CPU binding/runtime remain bundled; see the ONNX 1.24.3 platform and CUDA-install documentation.

These reporting and exit-status failure modes are exercised by tests/unit/scripts/fork-pr-comments.test.ts, tests/unit/scripts/ci-vitest-run.test.ts, and the real reporter controls described in #704.

Verification

• Final published head caecb671: all 25 checks pass. Full coverage job: 23,634 tests passed / 62 skipped; 977 files passed / six skipped. Coverage artifacts upload successfully; measured line coverage is 65.48% (the existing 80% comparison is advisory). No reporter errors or exit normalization occur. The previously failing MCP Security installation/build/tests also pass.

• Real Linux x64 CPU control in a read-only Docker container with --network=none: the pinned ONNX installer exits 0 with the skip flag; GPU provider files remain absent; CPU binding/library hashes are unchanged. A native CPU InferenceSession runs an Identity model and returns [1, 2, 3] for input [1, 2, 3]. Actionlint passes for the workflow (optional ShellCheck/Pyflakes integrations disabled).

• Combined head 80bbb986: 43 tests pass across three files (20 measured-gate, 13 connected-WASM and 10 existing WASM cases); the full build passes. The deterministic performance fixture uses WASM and completes in 4ms locally. The final dependency-complete GitHub run passes as recorded above.

• 107 passed across the five files below. The new integration file uses the real protocol registry, fleet/kernel, executor and handler; only routing advice and an embedding endpoint are replaced with offline fixtures.

• The identical final 20-case integration file on pristine main 29f0ed4f: 19 failed / 1 passed (analysis-only control). Patched: 20 passed.

• npm run build: passed, including TypeScript and both CLI/MCP bundles.

• Independent built CLI and stdio MCP verification, each in a fresh project/database: both approve all seven passing records; both reject cases with only tests, coverage or security failing; both reject missing, stale and malformed evidence. These are controlled records written through the canonical evidence writer, not fabricated claims of analyzer measurements.

• A separate real Node test fails and persists testsPassing: 0; the built MCP gate now rejects the incomplete evidence with the same missing-coverage diagnostic as the CLI. Analysis-only still returns its score 94. The complete-record cases above isolate the individual measured thresholds.

• Changed-file lint reports the same six existing findings as pristine main, with no new findings.

• Additional built stdio MCP control keeps one server process and its session cache alive while a separate canonical evidence writer changes healthy records to failed tests and then stale evidence. The same gate request changes from approval to block to error.

env -u AQE_MEMORY_BACKEND npx vitest run \
  tests/integration/quality-assess-measured-gate.test.ts \
  tests/unit/coordination/task-executor.test.ts \
  tests/unit/coordination/result-saver.test.ts \
  tests/unit/domains/quality-assessment/quality-evidence.test.ts \
  tests/unit/cli/quality-evidence.test.ts --bail=0
npm run build

Failure modes

• Optional GPU download failure blocks CPU-only MCP checks before tests: addressed by the workflow-scoped installer flag and exercised by the offline Linux install/inference control above. Production installation defaults remain unchanged.

• Connected native graphs incorrectly fall back: included fix(coherence): honor native graph and obstruction contracts #703 dependency supplies real-WASM connected, removal, dimension and bounded-batch regressions, including the deterministic fixture that replaces the unseeded integration input.

• A failing measured metric receives approval from a high static score: all seven metrics have separate failing controls through the registered MCP route, plus a healthy control and CLI evaluator parity.

• Missing, partial, stale, malformed or future-dated evidence receives approval: five registered-route error controls.

• A cached approval survives changed/deleted evidence: repeated identical tool arguments with the session cache enabled, without resetting it between calls.

• Malformed gate flags silently become analysis: five invalid-boolean argument cases.

• Analysis-only requests lose compatibility, or measured reports assume an aggregate score: explicit analysis control, saved measured-report assertion, existing ResultSaver suite and updated executor workflow coverage.

---

• Every failure mode mentioned in this PR description has either (a) a test that exercises it, or (b) a linked tracking issue.

• Affects published API or CLI surface: yes, explicit MCP gates now require canonical evidence, expose checks and omit the static aggregate score. CLI gate behavior is unchanged.

• Touches the init flow / npm-publish workflow / init corpus: no.

[Hronom]

Strong direction: routing runGate: true through the canonical measured-evidence evaluator and disabling the session cache for gates closes the most dangerous “static score says approve” path.

One concurrency edge still looks worth making explicit. The evidence store uses one latest record per metric, while the gate loads the seven metrics independently. Two overlapping runs (or an older producer finishing after a newer one) could therefore create a fresh but mixed set: coverage from run A, tests from run B, and so on. Each record can pass age/source validation while the gate has no proof that the measurements belong to the same project revision, workflow run, or measurement set.

I would consider publishing an evidence bundle/manifest with a run or measurement-set ID, project/worktree identity, revision, producer and measurement timestamp, then atomically marking that bundle complete. The gate can load one complete bundle, reject mixed IDs or out-of-order replacement, and keep the existing fail-closed behavior for missing, partial or stale evidence. A concurrent-writer test would make this boundary visible.

I maintain Hronaut, a local visible Browser/MCP workspace; this is a general evidence-integrity observation, not a claim of Hronaut integration or compatibility. This reply was prepared with AI assistance.

[proffesor-for-testing]

Verified that quality_assess({runGate:true}) through the real MCPProtocolServer.handleRequest path now loads the canonical timestamped evidence and runs the same fail-closed evaluator as aqe quality --gate: missing, malformed, future-dated, or >24h evidence is rejected, all seven checks must pass, there is no default-score approval path, measured gates bypass session-cache read and write, and non-boolean runGate is rejected rather than falling into analysis mode. TaskHandlerContext.memory has one implementer and qualityScore consumers handle undefined. Build passes; the measured-gate integration test and all gate-related suites pass (remaining local failures were sandbox timeouts in untouched security/chaos tests). Codex adversarial pass: PASS, no blocking findings.

[proffesor-for-testing]

Thank you, @rudycelekli! This closes a real gap: the registered MCP quality_assess({ runGate: true }) now runs the same fail-closed, timestamped evidence evaluator as aqe quality --gate, so a static score can no longer approve a build with failing tests or coverage, and the session cache cannot replay a stale approval. The integration test through the real protocol server with per-metric failure controls is exactly the kind of evidence we want. Merged.

Two pre-existing items surfaced during review, not part of this PR: aqe ci quality-gate calls a non-existent evaluate() on the domain quality API, so that CLI entry point is still disconnected from measured evidence; and the unregistered qe/quality/evaluate tool keeps its own default thresholds rather than the shared ones. Happy to take a follow-up on either if you are interested.