Skip to content

test(rules): mirror and cover PYD-013 - #179

Closed
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/pydantic-ai-ambiguous-tool-name
Closed

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/pydantic-ai-ambiguous-tool-name

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Engine half of a coordinated pair. Rules half: trustabl/agent-reliability-rules#107, on a branch of the same name, so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do.

What the pair adds

Ports the ambiguous-name check to this pack. CSDK-007, OAI-007, ADK-007, and MCP-003 all ship it; the newer packs had none. Same name_in list as CSDK-007.

What this PR does

  1. Mirrors pydantic_ai/tool_definition.yaml into testdata/rules-fixture/.
  2. Adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.
case expectation
def process(...) fires
def summarize_invoice(...) silent
def process_invoice_batch(...) silent

Three cases rather than two. The third is the one worth having: process_invoice_batch merely contains a listed word. It pins that name_in matches the whole name rather than a substring — the plausible regression, and the one that would otherwise flood well-named tools with findings across every pack using this list at once.

Verification

$ RULES_REPO=../trustabl-rules scripts/check-rules-sync.sh
rules fixture is in sync with production (86 files compared)

$ go vet ./internal/rules/
$ go test ./internal/rules/
ok  	github.com/trustabl/trustabl/internal/rules

Engine half of a coordinated pair with trustabl/agent-reliability-rules#107, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors pydantic_ai/tool_definition.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Three cases: the generic name, the verb-object remediation, and
process_invoice_batch — a name that merely contains a listed word and must
stay silent. The third pins that name_in matches the whole name rather
than a substring, which is the plausible regression and the one that would
otherwise flood well-named tools with findings.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI. The fixture copy of PYD-013 is byte identical to the production YAML in trustabl/agent-reliability-rules#107, which is what check-rules-sync.sh is looking for, and the three cases in policyRuleCases all do real work. parsePy sets ToolDef.Name from the function name and PredNameIn is a whole-name comparison, so process fires, summarize_invoice stays silent, and process_invoice_batch genuinely pins the substring boundary rather than passing vacuously. Adding that third case was the right call, since it is the regression that would matter across every pack sharing this list.

Nothing blocking on this side. Two coordination notes. #143 mirrors PYD-010 and PYD-011 into the same fixture file at the same offset and into the same policies_test.go hunk, so whichever of the two merges first leaves the other needing a rebase. And this half should land together with trustabl/agent-reliability-rules#107 rather than on its own.

I left a question on the rules half about the decorator name= override path, since buildTool takes the name from the function and ignores decorator kwargs. If we end up adding a guard for that, this file picks up a fourth case. The unrelated blank line dropped from policyRepoRuleCases is fine, just noting I saw it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants