Skip to content

test(rules): mirror and cover LC-024 - #176

Closed
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/langchain-ambiguous-tool-name
Closed

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/langchain-ambiguous-tool-name

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Engine half of a coordinated pair. Rules half: trustabl/agent-reliability-rules#104, on a branch of the same name, so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do.

What the pair adds

Ports the ambiguous-name check to this pack. CSDK-007, OAI-007, ADK-007, and MCP-003 all ship it; the newer packs had none. Same name_in list as CSDK-007.

What this PR does

  1. Mirrors langchain/tool_definition.yaml into testdata/rules-fixture/.
  2. Adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.
case expectation
def process(...) fires
def summarize_invoice(...) silent
def process_invoice_batch(...) silent

Three cases rather than two. The third is the one worth having: process_invoice_batch merely contains a listed word. It pins that name_in matches the whole name rather than a substring — the plausible regression, and the one that would otherwise flood well-named tools with findings across every pack using this list at once.

Verification

$ RULES_REPO=../trustabl-rules scripts/check-rules-sync.sh
rules fixture is in sync with production (86 files compared)

$ go vet ./internal/rules/
$ go test ./internal/rules/
ok  	github.com/trustabl/trustabl/internal/rules

Engine half of a coordinated pair with trustabl/agent-reliability-rules#104, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors langchain/tool_definition.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Three cases: the generic name, the verb-object remediation, and
process_invoice_batch — a name that merely contains a listed word and must
stay silent. The third pins that name_in matches the whole name rather
than a substring, which is the plausible regression and the one that would
otherwise flood well-named tools with findings.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI. The fixture copy is byte identical to the rules half in trustabl/agent-reliability-rules#104, and the base langchain/tool_definition.yaml already matches production, so there is no drift hiding underneath the addition and rules-sync will be satisfied once the two land together.

The three cases are right, and the third is the one I would have asked for. name_in is an exact case-insensitive match rather than a substring, so process_invoice_batch is exactly the case that pins that, and because parsePy builds ToolDef.Name from the function name, all three exercise the predicate for real rather than passing vacuously. The fire case is named process (on the list) and the silent cases are not, so both halves of what TestPolicyRules_AllRulesCovered requires are genuinely covered rather than just present.

The stray blank line dropped at the end of policyRepoRuleCases is a gofmt tidy, no objection to it riding along.

The only thing still outstanding is on the rules side: LC-024 needs a rationale doc in trustabl-rulebook before it ships. That one is our job rather than yours, and I am happy to pair on it. I will take these two as a pair.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants