Skip to content

test(rules): mirror and cover MCP-027, MCP-028 - #168

Closed
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/mcp-ts-description-quality
Closed

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/mcp-ts-description-quality

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Engine half of a coordinated pair. Rules half: trustabl/agent-reliability-rules#91, on a branch of the same name, so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do.

What the pair adds

The TypeScript counterpart to MCP-025/026. MCP-011 only checks that a description exists, so "TODO: describe this tool." and "Gets data." pass today while publishing no selection signal to connecting clients.

Same protocol-boundary argument as the Python pair — the consumer has no fallback context, the server's authors never observe the mis-selection, and the model chooses across every connected server from these strings alone — plus the TypeScript-specific point that the Zod input schema constrains the arguments once the model has chosen this tool, not whether choosing it was right.

What this PR does

  1. Mirrors mcp/tool_definition.yaml into testdata/rules-fixture/.
  2. Adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.
case expectation
MCP-027 — "TODO: describe this tool." fires
MCP-027 — real description silent
MCP-028 — "Gets data." fires
MCP-028 — full sentence silent
MCP-028 — empty description silent

Five cases rather than four. MCP-028 pairs description_length_lt: 40 with has_docstring: true so an empty description stays MCP-011's finding rather than double-reporting. The fifth case pins that guard.

Sequencing note: rules#88 / this repo's #165 (the MCP-025/026 Python pair) append to the same two files, so whichever pair lands second needs a trivial rebase in both repos — the rule blocks and test cases are independent and the resolution is "keep both." Happy to rebase on request.

Verification

$ RULES_REPO=../trustabl-rules scripts/check-rules-sync.sh
rules fixture is in sync with production (86 files compared)

$ go vet ./internal/rules/
$ go test ./internal/rules/
ok  	github.com/trustabl/trustabl/internal/rules

@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI. The fixture copy of mcp/tool_definition.yaml matches the production YAML in trustabl/agent-reliability-rules#91 exactly, which is the first thing I check on a cross repo pair, and the drive by removal of the stray blank line in policyRepoRuleCases makes the file gofmt clean again, so keep it.

The test cases hold up on a read. parseTSTool fails hard when discovery returns nothing, so the silent cases cannot pass vacuously, and the empty description case is a real test rather than a formality: the legacy tool() overload assigns the empty string literal to Description, so has_docstring is false and MCP-028 stays quiet while MCP-011 keeps the finding. Good call pinning that.

One suggestion, not a requirement. All five snippets use the legacy s.tool(name, description, schema, handler) overload while the existing MCP-011 cases use registerTool. Discovery fills Description on both paths, so nothing here is wrong, but one registerTool case would cover the shape most new servers are written in.

What is holding this up is coordination rather than code. MCP-027 and MCP-028 are also claimed by #135 and trustabl/agent-reliability-rules#100 for unrelated rules, so whichever pair lands first keeps the numbers and the other renumbers. A rulebook rationale doc is also needed before these ship, and we can take that or pair with you on it.

@sairenchristianbuerano

Copy link
Copy Markdown
Collaborator

Selected for merge. This pair keeps MCP-027 and MCP-028.

Closing trustabl/agent-reliability-rules#100 + #135, which claimed both ids for different rules (TypeScript idempotency and ambiguous tool name).

Several PRs claimed the same ids independently. We are resolving each id to a single pair rather than renumbering, so the competing PRs are being closed — the rules themselves were not rejected.

Merge the engine side first so a gap fails safe: a fixture for an unshipped rule is harmless, a rule with no fixture turns rules-sync red.

Engine half of a coordinated pair with trustabl/agent-reliability-rules#91, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors mcp/tool_definition.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Five cases rather than four. MCP-028 pairs description_length_lt with
has_docstring so an empty description stays MCP-011's finding instead of
double-reporting, and the fifth case pins that guard.

Note for sequencing: rules#88 (MCP-025/026, the Python pair) appends to
the same file, so whichever of the two pairs lands second needs a trivial
rebase in both repos — the rule blocks and the test cases are independent
and the resolution is "keep both".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants