feat(rules): consolidate the 36 open rule pack pull requests - #151
Merged
Merged
Conversation
CrewAI had no network-timeout rule, unlike Pydantic AI (PYD-006), AutoGen (AG2-012), and the Vercel AI SDK (VAI-011). The blast radius is larger here than in a single-agent SDK: a crew runs tasks in sequence and threads each task's output forward, so one stalled tool call blocks every task behind it. CREW-110's max_iter bounds reasoning steps, not the wall-clock of one tool call, so it never breaks the stall. Uses the same call_without_kwarg callee list as PYD-006.
…ontract LangChain had no error-contract rule; Claude SDK (CSDK-005), OpenAI (OAI-008), ADK (ADK-005), and MCP (MCP-006) all ship one. LangChain's default handle_tool_error=False lets a tool exception escape the AgentExecutor and abort the run, so the model is never told the step failed. Opting in with handle_tool_error=True only hands the model the raw str(exception): no failure mode, no retryability signal, and whatever internal detail the message carried. Python only. A LangChain.js counterpart is not shippable yet: PredHasRaise matches the Python grammar node "raise_statement", so has_raise is structurally always false for a TypeScript tool, whose throws parse as "throw_statement". A TS error-contract rule needs that engine-side gap closed first.
…ntract CrewAI had no error-contract rule; Claude SDK (CSDK-005), OpenAI (OAI-008), ADK (ADK-005), and MCP (MCP-006) all ship one. CrewAI stringifies an uncaught tool exception into the agent's observation, so the agent sees a bare message with no failure mode and no retryability signal, and usually burns its max_iter budget re-calling the tool the same way. A crew then threads that unusable output forward as context to every task sequenced behind it.
…ntract AutoGen had no error-contract rule; Claude SDK (CSDK-005), OpenAI (OAI-008), ADK (ADK-005), and MCP (MCP-006) all ship one. AutoGen runs tools on the executor agent's side and sends the result back as that agent's reply, so an uncaught exception either aborts initiate_chat or lands in the message history as a stringified traceback. In the second case both agents keep conversing with a traceback as the last observation, and the assistant re-issues the same call until AG2-004/AG2-006's round caps end the chat. The traceback also persists in the transcript, carrying internal detail into every later turn's context.
…lidation Pydantic AI had no path-safety rule; Claude SDK (CSDK-004), OpenAI (OAI-006), ADK (ADK-004), and MCP (MCP-005) all ship one. The SDK-specific point is that Pydantic AI's type layer does not cover this: a parameter annotated str or Path validates cleanly while still carrying ../../etc/passwd, because the annotation constrains the value's type and not the region of the filesystem it points at. The fix therefore pushes containment into the parameter's type via a validator rather than leaving it per call site. Uses the per-param call_uses_unnormalized_path_param, matching CSDK-004.
…tion CrewAI had no path-safety rule; Claude SDK (CSDK-004), OpenAI (OAI-006), ADK (ADK-004), and MCP (MCP-005) all ship one. The args_schema does not close the gap: a Pydantic field typed str or Path validates cleanly while still carrying ../../etc/passwd, because the schema constrains the argument's type and not the region of the filesystem it points at. Reach is wider than the owning agent — a crew shares one process and threads task output forward as context, so what this tool reads lands in the transcript every downstream task reasons over, and CREW-104 delegation lets an agent without the tool reach it through a peer. Uses the per-param call_uses_unnormalized_path_param, matching CSDK-004.
…tion AutoGen had no path-safety rule; Claude SDK (CSDK-004), OpenAI (OAI-006), ADK (ADK-004), and MCP (MCP-005) all ship one. The AutoGen-specific point is that a registered tool sidesteps the containment AG2-001 is about. Generated code may run inside a Docker executor, but a function registered with register_for_execution runs in the host process with the host's filesystem view regardless of how the code executor is configured, so the executor sandbox is not a mitigation here. Whatever the tool reads is then posted into the message history both agents keep reasoning over. Uses the per-param call_uses_unnormalized_path_param, matching CSDK-004.
LangChain had no idempotency rule; CrewAI (CREW-006), Pydantic AI (PYD-007), MCP (MCP-007), Claude SDK (CSDK-006), and OpenAI (OAI-009) all ship one. Repetition is routine in the LangChain agent loop rather than exceptional: an AgentExecutor step whose observation reads as inconclusive — a timeout, a truncated response, an error string returned under handle_tool_error — leads the model to re-reason about the same goal and re-issue the same action, with no way to know the first call already committed. with_retry and external retry wrappers compound it, since a network failure after the side effect landed looks identical to one before.
Ports the CSDK-017/018 description-quality pair to Pydantic AI. PYD-001 only checks that a docstring exists, so a tool whose docstring reads "TODO" or "Gets data." passes today while giving the model no selection signal. The cost is higher in this pack than in the Claude SDK, and the rule text says so: Pydantic AI also parses the docstring's parameter sections into the argument schema, so a placeholder or one-clause docstring strips the per-argument guidance at the same time as the tool-level guidance, and nothing in the pipeline reports the omission.
Ports the CSDK-017/018 description-quality pair to LangChain. LC-001 only checks that a docstring exists, so a tool whose docstring reads "TODO" or "Gets data." passes today while giving the model no selection signal. The framing is LangChain-specific on both counts. Mis-selection scales badly here because agents are routinely handed a dozen or more tools at once, and each mis-selected call burns an iteration against LC-102's max_iterations budget, so a run can exhaust its steps on tools it picked badly and return nothing useful. LC-018's fix also points at the @tool decorator's explicit description= for the case where the function name is genuinely ambiguous.
Ports the CSDK-017/018 description-quality pair to CrewAI. CREW-001 only checks that a docstring exists, so a tool whose docstring reads "TODO" or "Gets data." passes today while giving the agent no selection signal. The consequence travels further in a crew than in a single-agent SDK: an agent that picks the wrong tool produces a task output threaded forward as context to every task behind it, so one undescribed tool degrades work the owning agent never touches, and CREW-104 delegation lets peers reach the tool on the strength of the same placeholder. Each mis-selected call also spends one of the agent's CREW-110 max_iter steps.
Ports the CSDK-017/018 description-quality pair to AutoGen. AG2-007 only checks that a docstring exists, so a tool whose docstring reads "TODO" or "Gets data." passes today while giving the assistant no selection signal. How AutoGen fails on a bad guess is what makes this expensive: a wrongly chosen tool returns something that does not answer the assistant's intent, the assistant reads that as an inconclusive step and proposes another call, and the pair trades rounds until AG2-004's max_round or AG2-006's auto-reply cap ends the chat. The budget goes on a selection problem no error message points at. Both fixes name register_for_llm's description= as the clearer place to say it.
The Vercel AI pack had no path-safety rule; Claude SDK (CSDK-004/012), OpenAI (OAI-006), ADK (ADK-004), and MCP (MCP-005) all ship one. Mirrors CSDK-012, the TypeScript half of that pair, including its coarse-signal caveat: it flags any filesystem write rather than only unnormalized paths, because TS path-normalization analysis is not yet wired. The deployment shape is what makes it worth flagging here. Vercel AI tools typically run inside the same server process as the request handler rather than in a sandbox, so a model-steered write inherits the application's own filesystem permissions.
Ports OAI-010 to Pydantic AI. The loss is larger in this SDK than in the OpenAI pack, and the rule text says so: Pydantic AI emits OpenTelemetry spans for each run and tool call, so everything around a print is already correlated to a trace and the print is the one diagnostic landing outside it, unattached to the run that produced it. The fix names attaching the detail to the current span (Logfire or the OTel API) alongside the module logger.
Ports OAI-010 to LangChain. The loss is larger here than in the OpenAI pack, and the rule text says so: LangChain already threads every tool start, end, and error through its callback system into LangSmith or whichever tracer is configured, so a bare print is the one diagnostic landing outside that — detached from the run and the step that produced it, and simply absent the moment anyone debugs from a trace rather than a terminal.
Ports OAI-010 to CrewAI. CrewAI makes it worse than a lost log line: a crew running with verbose=True is already writing its own narration to stdout, so a tool print is interleaved into run commentary from several agents at once with nothing marking which agent, task, or tool call emitted it. What looks like working diagnostics in a terminal is unattributable the moment the crew runs anywhere else.
Ports OAI-010 to AutoGen. AutoGen makes it especially deceptive: the conversation itself is narrated to stdout, so a tool print lands mid transcript and reads as though it were part of the exchange, when it is invisible to the agents reasoning over it and carries no marker for which round or tool call produced it. Anyone reconstructing the run from a log sink rather than a terminal loses it entirely.
…nsport MCP had no observability rule. OpenAI ships OAI-010 and ADK ships ADK-009 for the same pattern, both at low severity as a lost-diagnostic problem. It is not a lost-diagnostic problem here, which is why this ships at medium rather than low. On a stdio server — the FastMCP default, and the usual way an editor or desktop client launches a server — stdout is the protocol channel, so a print interleaves with the newline-delimited JSON-RPC frames the client is parsing. The client hits a parse error on a line that is not JSON and drops the response or tears down the connection, and the symptom does not resemble the cause: a tool call that returns nothing, or a mid-session disconnect, traced back to a print that reads like harmless debugging. Confidence 0.7 rather than higher because a server run over HTTP or SSE escapes the corruption, though it still loses the diagnostic. The fix names stderr specifically, which the transport leaves alone.
Claude SDK and MCP were the two mature packs with no observability rule; OpenAI ships OAI-010 and ADK ships ADK-009 for the same pattern. Two Claude-SDK-specific consequences go beyond the lost log line OAI-010 describes. A tool defined in a Python SDK server is commonly served to the agent over an MCP stdio transport, where stdout carries the JSON-RPC frames and a loose print makes the client hit a parse error on a line that is not JSON. And when the SDK is driven programmatically, the host application is already reading the SDK's own message stream, so tool prints land interleaved with it rather than in the application's logs, attributable to no particular turn or tool call.
CSDK-003 covers the Python side; the TypeScript half was missing even though this pack ships TS rules throughout (CSDK-010..014, CSDK-120..131). OpenAI (OAI-016, OAI-024) and the Vercel AI SDK (VAI-011) already use has_http_call_without_timeout for exactly this. Node's fetch has no implicit deadline, so an unresponsive host stalls the conversation rather than failing it: the model gets no result and no error, and a max_turns cap does not help because the run is stuck inside one turn rather than taking too many. Compounds with CSDK-013 — a model-controlled URL that also cannot time out can be pointed at an internal host that never answers.
MCP-004 covers the Python side; the TypeScript half was missing even though the pack ships TS rules (MCP-011, MCP-013). OpenAI (OAI-016, OAI-024) and the Vercel AI SDK (VAI-011) already use has_http_call_without_timeout for exactly this. The stall crosses the server's trust boundary rather than staying local: the connecting client waits on a JSON-RPC response that never arrives, and MCP gives it no way to cancel an in-flight tool call, so it is left to its own timeout, an abandoned request, or a hung session, with nothing in the response saying why. On a stdio server a handler parked on a dead socket is holding the single process that serves the connection.
…no timeout ADK-003 covers the Python side; the TypeScript half was missing even though the pack ships TS rules (ADK-013, ADK-015, ADK-016, ADK-109). OpenAI (OAI-016, OAI-024) and the Vercel AI SDK (VAI-011) already use has_http_call_without_timeout for exactly this. The composition ADK encourages makes the stall worse rather than better: inside a SequentialAgent a stalled tool blocks every step after it, and inside a ParallelAgent the fan-out waits on its slowest branch, so one unresponsive host stalls work unrelated to it. ADK-108's max_iterations bounds how many times a LoopAgent goes round, not how long one tool call may run, so no configured limit breaks the stall. Numbered ADK-114 to leave room for ADK-111 in the open PR #51 and ADK-112 in my PR #79.
Ports the CSDK-017/018 description-quality pair to the OpenAI Agents SDK. OAI-001 only checks that a docstring exists, so a tool whose docstring reads "TODO" or "Gets data." passes today while giving the model no selection signal. Two OpenAI-specific points. Strict schemas do not compensate: OAI-003's strict_mode constrains the shape of the arguments, never whether calling this tool was the right move, so a placeholder description produces a well-formed call to the wrong tool. And with handoffs the cost is not just a wrong call — an agent picks between its own tools and its peers' on the strength of these strings, so a thin description can route the whole conversation to the wrong agent. Mis-selection also spends a turn against the OAI-112 max_turns budget with nothing in a trace attributing it to the description.
Ports the CSDK-017/018 description-quality pair to MCP. MCP-001 only checks that a docstring exists, so a tool whose docstring reads "TODO" or "Gets data." passes today while giving connecting clients no selection signal. A placeholder costs more across a protocol boundary than inside an in-process SDK. The consumer has no other context to fall back on — it cannot read the source, the neighboring tools, or the project's docs the way a developer in the repo can. The server's authors also never see the consequence, since mis-selection surfaces in someone else's client session as a wrong answer rather than as an error on this side. And a client typically connects several servers at once, so the model chooses across all of them from these strings alone and a thin description competes badly against a well-described tool from an unrelated server.
Ports the CSDK-017/018 description-quality pair to the Google ADK. ADK-001 only checks that a docstring exists, so a tool whose docstring reads "TODO" or "Gets data." passes today while giving the model no selection signal. Two ADK-specific points. The ADK parses the docstring for the per-parameter descriptions that accompany the generated schema, so a placeholder or one-clause docstring strips argument-level guidance at the same time as tool-level guidance. And in an agent tree the mis-selection does not stay local: a tool picked wrongly inside one branch produces output the following agents treat as established fact, so the error is laundered into shared session state rather than surfacing where it began. ADK-102/107's before_tool_callback can block a call it recognizes as wrong but cannot supply the judgment the description was meant to give. Numbered ADK-115/116 to leave room for ADK-111 in the open PR #51 and the IDs used by my other open PRs.
…ty rules CSDK-017/018 cover the Python side; the TypeScript half was missing. CSDK-014 only checks that a description exists, so a tool whose description reads "TODO: describe this tool." or "Gets data." passes today while leaving the tool in exactly the state CSDK-014 exists to prevent. The gap costs more on the TypeScript side than in Python, and the rule text says so: CSDK-014's own explanation notes there is no docstring fallback, so the description argument is the entire prompt-side account of the tool. The Zod input schema does not compensate — it constrains the shape of the arguments once the model has decided to call this tool, and says nothing about whether calling it was the right move.
The TypeScript counterpart to MCP-025/026. MCP-011 only checks that a description exists, so a registration described "TODO: describe this tool." or "Gets data." passes today while publishing no selection signal to connecting clients. Same protocol-boundary argument as the Python pair — the consumer has no fallback context, the server's authors never observe the mis-selection, and the model chooses across every connected server from these strings alone — plus the TypeScript-specific point that the Zod input schema constrains the arguments once the model has chosen this tool, not whether choosing it was right.
… rules The TypeScript counterpart to OAI-025/026. OAI-022 only checks that a description exists, so a tool described "TODO: describe this tool." or "Gets data." passes today while leaving the tool in exactly the state OAI-022 exists to prevent. The Zod parameters schema does not compensate: it constrains the shape of the arguments once the model has decided to call this tool, never whether that decision was right, so a placeholder yields a well-formed call to the wrong tool. With handoffs the reach is wider — an agent picks between its own tools and its peers' from these strings, so the stub can route the conversation to the wrong agent rather than merely the wrong function.
The TypeScript counterpart to LC-018/019. LC-010 only checks that a description exists, so a tool described "TODO: describe this tool." or "Gets data." passes today while giving the model no selection signal. LangChain.js has no docstring to fall back on, so the description field is the entire account of the tool the model sees, and the Zod schema does not compensate — it constrains the arguments once the model has chosen this tool, never whether choosing it was right. Mis-selection scales badly here because an agent is routinely handed a dozen or more tools at once, and each wrong pick spends an iteration against the maxIterations bound LC-111 checks for.
MCP-005 covers the Python path-safety case; the TypeScript half was missing. Mirrors CSDK-012, including its coarse-signal caveat — it flags any filesystem write rather than only unnormalized paths, because TS path-normalization analysis is not yet wired. Deployment is what sharpens this for MCP. A stdio server is launched as a subprocess by whatever client the user is running, so it inherits that user's own filesystem permissions rather than a service account's, and a write escaping its intended directory reaches the user's home directory, dotfiles, and SSH keys. The server also cannot see the injection: it receives a well-formed tools/call for a path it has no way to distinguish from a legitimate one.
OAI-006 covers the Python path-safety case; the TypeScript half was missing. Mirrors CSDK-012, including its coarse-signal caveat — it flags any filesystem write rather than only unnormalized paths, because TS path-normalization analysis is not yet wired. The guardrail story does not cover this, which is the point worth making in an SDK built around guardrails: OAI-101 concerns input guardrails on the agent, which screen what enters the conversation, not what a tool does with an argument once the model has produced it, and a call reaching execute() has already passed whatever guardrails were configured. Tools here also typically run in the same server process as the request handler rather than a sandbox.
…filesystem ADK-004 covers the Python path-safety case; the TypeScript half was missing. Mirrors CSDK-012, including its coarse-signal caveat — it flags any filesystem write rather than only unnormalized paths, because TS path-normalization analysis is not yet wired. The callback gate does not close this by default: ADK-102 and ADK-107 are about before_tool_callback being present at all, and a callback that is present still has to inspect the path itself and know which root is allowed. Being wired up is not the same as containing anything. Agent composition widens the reach too — any branch listing this tool can reach it, and a write performed in one branch is visible to every agent reading the shared session state afterward, so the fix belongs inside the tool.
Ports the ambiguous-name check to langchain. Claude SDK (CSDK-007), OpenAI (OAI-007), ADK (ADK-007), and MCP (MCP-003) all ship it; the five newer packs had none. The tool name sits directly beside the description in what the model sees, so it is half the selection signal, and a generic verb like process or handle spends that half on nothing.
Ports the ambiguous-name check to crewai. Claude SDK (CSDK-007), OpenAI (OAI-007), ADK (ADK-007), and MCP (MCP-003) all ship it; the five newer packs had none. The tool name sits directly beside the description in what the model sees, so it is half the selection signal, and a generic verb like process or handle spends that half on nothing.
Ports the ambiguous-name check to autogen. Claude SDK (CSDK-007), OpenAI (OAI-007), ADK (ADK-007), and MCP (MCP-003) all ship it; the five newer packs had none. The tool name sits directly beside the description in what the model sees, so it is half the selection signal, and a generic verb like process or handle spends that half on nothing.
Ports the ambiguous-name check to pydantic_ai. Claude SDK (CSDK-007), OpenAI (OAI-007), ADK (ADK-007), and MCP (MCP-003) all ship it; the five newer packs had none. The tool name sits directly beside the description in what the model sees, so it is half the selection signal, and a generic verb like process or handle spends that half on nothing.
trustabl
approved these changes
Sep 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates 36 of @bradAGI's open rule pull requests into one branch so they can merge together with their engine fixtures.
Why this is one pull request
@bradAGI deleted the
bradAGI/trustablfork, which disables GitHub's Reopen button on all 36 engine pull requests. The commits survive in this repository's ownrefs/pull/<n>/head, so every one was recovered unchanged and is still authored by @bradAGI — nothing was rewritten or re-done.They are consolidated into a single branch rather than 36 because
rules-syncfails in both directions — a rule with no fixture and a fixture with no rule — so 36 separate pairs means 36 chances to leavemainred. Two pull requests merged back to back means one short window.Verified
trustabl rules validate: 275 rules valid under schema 16, up from 228go build ./...clean,gofmtcleango test ./internal/rules/: 681 policy cases pass, 0 failcheck-rules-sync.shclean between the two branchesMerge order
Merge trustabl/agent-reliability-analyzer#218 and this one back to back.
Closes these once merged
The 36 pull requests below are contained in this branch, commit for commit, and should be closed when this merges: