Skip to content

feat(langchain): add LC-024, ambiguous tool name - #104

Open
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/langchain-ambiguous-tool-name
Open

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/langchain-ambiguous-tool-name

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Ports the ambiguous-name check to LangChain. Claude SDK (CSDK-007), OpenAI (OAI-007), ADK (ADK-007), and MCP (MCP-003) all ship it; the newer packs had none.

The tool name sits directly beside the description in what the model sees, so it is half the selection signal — and a generic verb like process or handle spends that half on nothing. A LangChain agent is routinely handed a dozen or more tools at once, so a name that does not discriminate competes against every neighbor in the list — and each wrong pick spends an iteration against LC-102's max_iterations budget.

Same name_in list as CSDK-007, so behavior stays consistent across packs.

Verification — engine built at main:

$ trustabl rules validate .
OK: 85 rule pack(s), 211 rule(s) valid under rule schema version 14

Fire (tool named process): includes LC-024
Silent (same tool renamed summarize_invoice): LC-024 absent

A note on scope. I drafted a fifth rule in this batch for the Vercel AI SDK and dropped it after testing: ToolDef.Name is deliberately empty for vercel_ai_tool, because the SDK derives a tool's name from the key in the tools: { ... } record rather than from the tool({...}) call. The engine asserts this directly —

// internal/analysis/ts_vercel_tools_test.go
if tl.Name != "" {
    t.Errorf("Name: got %q, want empty (Vercel derives name from the tools-record key)", tl.Name)
}

so every name-based predicate (name_in, name_has_prefix) is structurally inert for that pack — it also blocks a Vercel idempotency rule, which is why the pack has none. Back-filling Name from the record key at discovery time would unblock both. Happy to open that engine PR if it'd be useful.

Paired engine PR follows, mirroring the rule into testdata/rules-fixture/ with fire/silent cases.

Ports the ambiguous-name check to langchain. Claude SDK (CSDK-007), OpenAI
(OAI-007), ADK (ADK-007), and MCP (MCP-003) all ship it; the five newer
packs had none.

The tool name sits directly beside the description in what the model sees,
so it is half the selection signal, and a generic verb like process or
handle spends that half on nothing.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI, this is a clean port. LC-024 matches CSDK-007 exactly on the name_in list, severity, and confidence, which is what I want for a list that is now shared across five packs, and langchain_tool is a valid applies_to at scope: tool with name_in already in the schema, so nothing here needs a schema_version bump. The LC-102 cross reference checks out (that is the max_iterations rule), and LC-024 is the only free number left in the LC tool range given the other LangChain PRs in flight, so no collision to sort out.

Two follow-ups rather than blockers. LangChain.js tools do carry a name (buildTSLangChainTool reads the config object's name literal), so a language: typescript twin of this rule would work where the Vercel one structurally could not. And yes, please open the engine PR that back-fills ToolDef.Name from the Vercel tools-record key.

One nit: the last sentence of explanation is a single long unwrapped line, so please wrap it to match the pack. Also, your rules validate output reports 211 rules where main plus this one comes to 207, was that run against a tree carrying your other branches?

Before this reaches users it needs a rationale doc in trustabl-rulebook. That part is on us, and I am happy to pair on it. I will take this together with trustabl/agent-reliability-analyzer#176.

ivanpaghubasan pushed a commit to ivanpaghubasan/agent-reliability-analyzer that referenced this pull request Sep 22, 2026
Engine half of a coordinated pair with trustabl/agent-reliability-rules#104, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors langchain/tool_definition.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Three cases: the generic name, the verb-object remediation, and
process_invoice_batch — a name that merely contains a listed word and must
stay silent. The third pins that name_in matches the whole name rather
than a substring, which is the plausible regression and the one that would
otherwise flood well-named tools with findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants