Skip to content

test(rules): mirror and cover MCP-025, MCP-026 - #165

Closed
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/mcp-description-quality
Closed

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/mcp-description-quality

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Engine half of a coordinated pair. Rules half: trustabl/agent-reliability-rules#88, on a branch of the same name, so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do.

What the pair adds

Ports the CSDK-017/018 description-quality pair to MCP. MCP-001 only checks that a docstring exists, so """TODO: describe this tool.""" and """Gets data.""" pass today.

A placeholder costs more across a protocol boundary than inside an in-process SDK, for three reasons the rule text makes explicit: the consumer has no fallback context (it can't read the source or neighboring tools); the server's authors never observe the mis-selection, since it surfaces in someone else's client session; and a client typically connects several servers at once, so the model picks across all of them from these strings alone.

What this PR does

  1. Mirrors mcp/tool_definition.yaml into testdata/rules-fixture/.
  2. Adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.
case expectation
MCP-025 — """TODO: describe this tool.""" fires
MCP-025 — real description silent
MCP-026 — """Gets data.""" fires
MCP-026 — full sentence silent
MCP-026 — no docstring at all silent

Five cases rather than four. MCP-026 pairs description_length_lt: 40 with has_docstring: true so an absent docstring stays MCP-001's finding rather than double-reporting. The fifth case pins that guard.

Verification

$ RULES_REPO=../trustabl-rules scripts/check-rules-sync.sh
rules fixture is in sync with production (86 files compared)

$ go vet ./internal/rules/
$ go test ./internal/rules/
ok  	github.com/trustabl/trustabl/internal/rules

Engine half of a coordinated pair with trustabl/agent-reliability-rules#88, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors mcp/tool_definition.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Five cases rather than four. MCP-026 pairs description_length_lt with
has_docstring so an absent docstring stays MCP-001's finding instead of
double-reporting — an empty description is length 0, which is also under
the threshold — and the fifth case is what pins that guard.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI. This is a well put together pair. The fixture copy of mcp/tool_definition.yaml is byte identical to the production half in trustabl/agent-reliability-rules#88, both rules reuse predicates that already exist on main (has_description_text, has_docstring, description_length_lt), and MCP-025/MCP-026 land in a free slot (MCP-023 and MCP-024 are claimed by other open PRs, but nothing here collides).

The five cases are the right shape. Reading them against the match expressions: the placeholder case trips the todo needle, the stub case is 10 characters against the 40 threshold, and the absent-docstring case genuinely pins the has_docstring: true guard rather than passing vacuously, which is the reason that guard is there.

Two small things. The diff also removes an unrelated blank line at the end of policyRepoRuleCases; please drop that hunk, since this file is heavily contended right now and every extra hunk is conflict surface. Separately, a note for the rationale doc rather than the YAML: has_description_text is a case-insensitive substring match, so a task-management server with "Add a todo item to the list." will fire MCP-025. Worth naming under known gaps.

Both rules still need a rulebook rationale doc before they ship, and we can handle or pair on that. Merge this together with trustabl/agent-reliability-rules#88, since rules-sync fails if either half lands alone. Happy to take it once the stray hunk is gone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants