Skip to content

test(rules): mirror and cover LC-021, LC-022 - #171

Closed
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/langchain-ts-description-quality
Closed

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/langchain-ts-description-quality

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Engine half of a coordinated pair. Rules half: trustabl/agent-reliability-rules#93, on a branch of the same name, so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do.

What the pair adds

The TypeScript counterpart to LC-018/019. LC-010 only checks that a description exists, so "TODO: describe this tool." and "Gets data." pass today. Three points, in the order they bite:

  • No docstring fallback in LangChain.js — the description field is the entire account of the tool the model sees.
  • The Zod schema doesn't compensate — it constrains the arguments once the model has chosen this tool, not whether choosing it was right.
  • Mis-selection scales badly and costs the run — agents are routinely handed a dozen or more tools at once, and each wrong pick spends an iteration against the maxIterations bound LC-111 checks for.

What this PR does

  1. Mirrors langchain/tool_definition.yaml into testdata/rules-fixture/.
  2. Adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.
case expectation
LC-021 — "TODO: describe this tool." fires
LC-021 — real description silent
LC-022 — "Gets data." fires
LC-022 — full sentence silent
LC-022 — no description field at all silent

Five cases rather than four. LC-022 pairs description_length_lt: 40 with has_docstring: true so an absent description stays LC-010's finding rather than double-reporting. The fifth case pins that guard.

Sequencing note: rules#66 / this repo's #145 (the LC-018/019 Python pair) append to the same two files, so whichever pair lands second needs a trivial rebase in both repos — resolution is "keep both." Happy to rebase on request.

Verification

$ RULES_REPO=../trustabl-rules scripts/check-rules-sync.sh
rules fixture is in sync with production (86 files compared)

$ go vet ./internal/rules/
$ go test ./internal/rules/
ok  	github.com/trustabl/trustabl/internal/rules

Engine half of a coordinated pair with trustabl/agent-reliability-rules#93, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors langchain/tool_definition.yaml into testdata/rules-fixture and
adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered
requires.

Five cases rather than four. LC-022 pairs description_length_lt with
has_docstring so an absent description stays LC-010's finding instead of
double-reporting, and the fifth case pins that guard.

Note for sequencing: rules#66 (LC-018/019, the Python pair) appends to the
same file, so whichever of the two pairs lands second needs a trivial
rebase in both repos — the rule blocks and the test cases are independent
and the resolution is "keep both".
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI. This is the half that makes trustabl/agent-reliability-rules#93 safe to land, and the mirror is exact: the fixture copy and the production file end on the same blob hash, so check-rules-sync.sh has nothing to flag.

The five cases are real rather than vacuous. Walking each one against origin/main: the @langchain/core/tools import routes tool(fn, {...}) into DiscoverTSLangChainTools, description is lifted into ToolDef.Description, so "TODO: describe this tool." trips has_description_text while the full replacement contains none of the needles, and "Gets data." sits under the 40 character bound while the sentence-length one clears it. The fifth case, an absent description staying silent on LC-022, is the one I most wanted to see, since it pins the has_docstring: true guard that keeps this from double reporting against LC-010.

Your note is right that trustabl/agent-reliability-rules#66 and #145 append to the same two files, so whichever pair lands second needs the trivial rebase, and keep-both is the resolution.

The only thing still outstanding is a rulebook rationale doc for the two new rules, which is on us rather than on you. Both halves look good to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants