Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
f36b855
feat(crewai): add CREW-007, tool network call has no timeout
bradAGI Aug 24, 2026
0d6c6ad
feat(langchain): add LC-008, tool raises without a structured error c…
bradAGI Aug 24, 2026
ccccc11
feat(crewai): add CREW-008, tool raises without a structured error co…
bradAGI Aug 24, 2026
8b7830c
feat(autogen): add AG2-013, tool raises without a structured error co…
bradAGI Aug 24, 2026
4384de0
feat(pydantic_ai): add PYD-009, path parameter used in I/O without va…
bradAGI Aug 24, 2026
06eea6d
feat(crewai): add CREW-009, path parameter used in I/O without valida…
bradAGI Aug 24, 2026
deaa355
feat(autogen): add AG2-014, path parameter used in I/O without valida…
bradAGI Aug 24, 2026
4d69fd9
feat(langchain): add LC-017, mutating tool has no idempotency key
bradAGI Aug 24, 2026
7ae93ff
feat(pydantic_ai): add PYD-010, PYD-011 tool description quality rules
bradAGI Aug 24, 2026
4c6a00f
feat(langchain): add LC-018, LC-019 tool description quality rules
bradAGI Aug 24, 2026
8b168d7
feat(crewai): add CREW-010, CREW-011 tool description quality rules
bradAGI Aug 24, 2026
b3ceba7
feat(autogen): add AG2-016, AG2-017 tool description quality rules
bradAGI Aug 24, 2026
c254b5d
feat(vercel_ai): add VAI-015, tool writes to the filesystem
bradAGI Aug 24, 2026
bfa7683
feat(pydantic_ai): add PYD-012, tool prints to stdout for diagnostics
bradAGI Aug 24, 2026
5b77642
feat(langchain): add LC-020, tool prints to stdout for diagnostics
bradAGI Aug 24, 2026
bd02920
feat(crewai): add CREW-012, tool prints to stdout for diagnostics
bradAGI Aug 24, 2026
7969977
feat(autogen): add AG2-018, tool prints to stdout for diagnostics
bradAGI Aug 24, 2026
d22de8f
feat(mcp): add MCP-023, tool prints to stdout, corrupting a stdio tra…
bradAGI Aug 24, 2026
5b91673
feat(claude_sdk): add CSDK-019, tool prints to stdout for diagnostics
bradAGI Aug 24, 2026
9e54e8c
feat(claude_sdk): add CSDK-020, TypeScript tool HTTP call has no timeout
bradAGI Aug 24, 2026
c373da0
feat(mcp): add MCP-024, TypeScript tool HTTP call has no timeout
bradAGI Aug 24, 2026
76a5ab6
feat(google_adk): add ADK-114, TypeScript FunctionTool HTTP call has …
bradAGI Aug 24, 2026
10539b5
feat(openai_sdk): add OAI-025, OAI-026 tool description quality rules
bradAGI Aug 24, 2026
5125536
feat(mcp): add MCP-025, MCP-026 tool description quality rules
bradAGI Aug 24, 2026
c8571ed
feat(google_adk): add ADK-115, ADK-116 tool description quality rules
bradAGI Aug 24, 2026
669c0a9
feat(claude_sdk): add CSDK-021, CSDK-022 TypeScript description quali…
bradAGI Aug 24, 2026
735821d
feat(mcp): add MCP-027, MCP-028 TypeScript description quality rules
bradAGI Aug 24, 2026
4e71c7c
feat(openai_sdk): add OAI-027, OAI-028 TypeScript description quality…
bradAGI Aug 24, 2026
00b9e65
feat(langchain): add LC-021, LC-022 TypeScript description quality rules
bradAGI Aug 24, 2026
52fc628
feat(mcp): add MCP-029, TypeScript tool writes to the filesystem
bradAGI Aug 24, 2026
49eaa50
feat(openai_sdk): add OAI-029, TypeScript tool writes to the filesystem
bradAGI Aug 24, 2026
148cba3
feat(google_adk): add ADK-117, TypeScript FunctionTool writes to the …
bradAGI Aug 24, 2026
8f4d5b5
feat(langchain): add LC-024, ambiguous tool name
bradAGI Aug 24, 2026
266d725
feat(crewai): add CREW-013, ambiguous tool name
bradAGI Aug 24, 2026
9c98e05
feat(autogen): add AG2-019, ambiguous tool name
bradAGI Aug 24, 2026
852706e
feat(pydantic_ai): add PYD-013, ambiguous tool name
bradAGI Aug 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions autogen/error_handling.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
policy:
id: autogen_error_handling
name: AutoGen error contract hygiene
category: autogen
description: >
Rules covering how a registered AutoGen tool surfaces failure. An
unhandled exception in a tool breaks the two-agent reply loop the whole
conversation is built on, so the failure surfaces as a stalled or aborted
chat rather than something either agent can reason about.

rules:
- id: AG2-013
title: AutoGen tool raises without a structured error contract
severity: low
confidence: 0.6
language: python
applies_to:
- autogen_tool
scope: tool
match:
all:
- has_raise: true
- has_try_except: false
explanation: >
This registered tool raises an exception it never catches. AutoGen
executes tools on the executor agent's side of the conversation and sends
the result back as that agent's reply, so an uncaught exception either
aborts initiate_chat outright or comes back as a stringified traceback
posted into the message history. In the second case both agents keep
talking with a traceback as the last observation: the assistant has no
failure mode to branch on and no signal about whether a retry could
succeed, so it re-issues the same call until the max_round or auto-reply
cap (AG2-004, AG2-006) ends the conversation. The traceback also stays in
the transcript, carrying file paths and internal detail into every
subsequent turn's context.
fix: >
Catch the failure modes you expect and return a structured value the
assistant can branch on — name the failure and say whether retrying could
help and what would need to change — instead of letting the exception
escape the registered function. Reserve raising for programmer errors no
amount of agent reasoning can work around.
39 changes: 39 additions & 0 deletions autogen/observability.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
policy:
id: autogen_observability
name: AutoGen tool observability hygiene
category: autogen
description: >
Rules covering how a registered AutoGen tool emits diagnostics. A tool body
that prints to stdout writes into the same stream the conversation itself is
narrated on, so the record is neither visible to the agents nor separable
from the transcript around it.

rules:
- id: AG2-018
title: AutoGen tool prints to stdout for diagnostics
severity: low
confidence: 0.65
language: python
applies_to:
- autogen_tool
scope: tool
match:
has_print_call: true
explanation: >
The registered tool's body calls print(), which writes to the process's
stdout. Neither agent sees it — only the return value is posted back into
the conversation as the executor's reply — so the output silently
disappears in any deployment that captures structured records rather than
raw stdout. AutoGen makes that especially deceptive: the conversation
itself is narrated to stdout, so a tool print lands in the middle of the
transcript and reads as though it were part of the exchange, when in fact
it is invisible to the agents reasoning over it and carries no marker for
which round or which tool call produced it. Anyone reconstructing the run
from a log sink rather than a terminal loses it entirely.
fix: >
Remove the print(). For operator diagnostics, emit through a module logger
(logging.getLogger(__name__).info(...)) so the record carries its own
module and level and lands in the application's log sink rather than the
conversation narration. If the information needs to reach the assistant,
return it as part of the tool's result, which is the only channel the
agents actually read.
50 changes: 50 additions & 0 deletions autogen/path_safety.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
policy:
id: autogen_path_safety
name: AutoGen filesystem path safety
category: autogen
description: >
Rules that catch a model-supplied filesystem path flowing into an I/O call
inside a registered AutoGen tool without containment. A traversal payload
like ../../etc/passwd reaches real filesystem state if the tool does not
resolve the path and check it against an allowed root.

rules:
- id: AG2-014
title: AutoGen tool uses a path parameter in I/O without validation
severity: high
confidence: 0.7
language: python
applies_to:
- autogen_tool
scope: tool
# call_uses_unnormalized_path_param is per-param: a tool with two path
# params and one .resolve() correctly fires on the unresolved one. A
# body-wide check would suppress the finding whenever any normalization
# appears, masking real exposure on the other param.
match:
call_uses_unnormalized_path_param:
callees:
- open
- Path
callee_prefixes:
- shutil.
- os.
explanation: >
This registered tool takes a path-like parameter and hands it to a file or
directory operation without resolving it or checking it against an allowed
root, so a model-supplied ../../etc/passwd reaches the real filesystem.
Type hints do not close this: a parameter annotated str or Path passes
AutoGen's schema generation while still carrying a traversal payload,
because the annotation constrains the argument's type and not the region
of the filesystem it points at. Registered tools also sidestep the
containment AG2-001 is about — a crew may run its generated code inside a
Docker executor, but a tool registered with register_for_execution runs in
the host process, with the host's filesystem view, whatever the code
executor is configured to do. Whatever the tool reads is then posted into
the message history both agents keep reasoning over.
fix: >
Resolve the path with Path(...).resolve() and assert the result sits under
an allowed root before touching it, rejecting anything that escapes. Keep
that check inside the registered function itself: it is the only boundary
the tool path crosses, since the code executor's sandbox does not apply
here.
92 changes: 92 additions & 0 deletions autogen/tool_definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,95 @@ rules:
Annotate every parameter with a concrete type (str, int, list[str], a
typing.Annotated description, etc.). AutoGen propagates these annotations into
the schema the model sees, so the model emits correctly-shaped arguments.

- id: AG2-016
title: AutoGen tool description is a placeholder
severity: low
confidence: 0.85
language: python
applies_to:
- autogen_tool
scope: tool
match:
has_description_text:
- todo
- tbd
- fixme
- placeholder
- no description
- does stuff
explanation: >
The docstring passes the AG2-007 has-a-description check but carries a
placeholder marker instead of real content. AutoGen advertises the tool to
the assistant through this text, so a stub like "TODO: describe this tool"
is functionally indistinguishable from no description at all and the
assistant has to guess from the function name. The way AutoGen fails on a
bad guess is what makes this expensive: a wrongly chosen tool returns
something that does not answer the assistant's intent, the assistant reads
that as an inconclusive step and proposes another call, and the pair
trades rounds until AG2-004's max_round or AG2-006's auto-reply cap ends
the chat — burning the conversation budget on a selection problem no error
message points at.
fix: >
Replace the placeholder with a real description covering what the tool
does, what it returns, and when the assistant should call it rather than a
neighboring tool. Passing an explicit description= to register_for_llm
overrides the docstring and is the clearer place to say it.

- id: AG2-017
title: AutoGen tool description is too short to guide model selection
severity: low
confidence: 0.8
language: python
applies_to:
- autogen_tool
scope: tool
match:
all:
- has_docstring: true
- description_length_lt: 40
explanation: >
A description under 40 characters is rarely enough to convey what a tool
does, what it returns, and when to call it rather than a similarly named
neighbor. AutoGen passes this text to the assistant as its only selection
signal, so a stub like "Gets data." leaves scope and preconditions to
guesswork. Each mis-selected call costs a full round of the conversation —
the assistant proposes, the executor runs, the result comes back unhelpful
— so a thin description spends the max_round budget rather than the run
producing an answer.
fix: >
Expand the description to at least a full sentence covering inputs,
outputs, and the situation in which this tool should be used over the
alternatives, either in the docstring or via register_for_llm's
description= argument.

- id: AG2-019
title: Ambiguous AutoGen tool name
severity: low
confidence: 0.9
language: python
applies_to:
- autogen_tool
scope: tool
match:
name_in:
- process
- handle
- run
- do
- execute
- perform
- work
- go
- thing
- stuff
explanation: >
The tool name is a generic verb that says nothing about what the tool
acts on. The name sits directly beside the description in what the model
sees, so it is half the selection signal, and a name like process or
handle spends that half on nothing — the model either calls the tool for
the wrong job or passes over it entirely. Each wrong pick costs a full round of the conversation — propose, execute, reply — so an ambiguous name spends the AG2-004 max_round budget rather than the chat reaching an answer.
fix: >
Rename to a verb-object form that names the thing acted on:
summarize_invoice, refund_charge, fetch_order_status. Keep the
description for the detail and let the name carry the subject.
31 changes: 31 additions & 0 deletions claude_sdk/network.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,34 @@ rules:
fix: >
Pass `timeout=` (typically 5–30s) to the request. Surface failures as a
structured error the model can react to.

- id: CSDK-020
title: TypeScript Claude SDK tool HTTP call has no timeout
severity: high
confidence: 0.6
language: typescript
applies_to:
- claude_sdk_tool
scope: tool
match:
has_http_call_without_timeout: true
explanation: >
This TypeScript Claude SDK tool makes an outbound HTTP call (fetch /
axios / got / undici) with no deadline — no signal, timeout, or
abortSignal option. Node's fetch has no implicit deadline, so a slow or
unresponsive host blocks the tool callback until the socket eventually
dies. Because the call sits inside the agent's turn, that stalls the
conversation rather than failing it: the model gets no result and no
error, the turn cannot advance, and a max_turns cap does not help because
the run is stuck inside a single turn rather than taking too many. In a
server embedding the SDK it also holds the request worker for the
duration. The exposure compounds with CSDK-013: a tool that fetches a
model-controlled URL and cannot time out can be pointed at an internal
host that simply never answers.
fix: >
Attach a deadline. On modern runtimes,
await fetch(url, { signal: AbortSignal.timeout(15_000) }); on older ones,
create an AbortController, abort it from a setTimeout, pass
controller.signal, and clear the timer in a finally. axios and got take a
timeout option directly. Return the abort as a structured tool error so
the model can retry or route around it instead of the turn hanging.
41 changes: 41 additions & 0 deletions claude_sdk/observability.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
policy:
id: claude_sdk_observability
name: Claude Agent SDK tool observability hygiene
category: claude_sdk
description: >
Rules covering how a Claude Agent SDK tool emits diagnostics. A tool body
that prints to stdout writes somewhere neither the model nor the
application's log sink reads, and can corrupt a transport that uses stdout
as a protocol channel.

rules:
- id: CSDK-019
title: Tool prints to stdout for diagnostics
severity: low
confidence: 0.65
language: python
applies_to:
- claude_sdk_tool
scope: tool
match:
has_print_call: true
explanation: >
The tool body calls print(), which writes diagnostics to the process's
stdout. The model never sees that output — only the tool's return value
flows back into the agent loop — so the print silently disappears in any
deployment that captures structured records rather than raw stdout. Two
Claude-SDK-specific consequences make it worse than a lost log line. A
tool defined in a Python SDK server is commonly served to the agent over
an MCP stdio transport, where stdout carries the JSON-RPC frames: a loose
print interleaves with them and the client hits a parse error on a line
that is not JSON. And when the SDK is driven programmatically, the host
application is reading the SDK's own message stream, so tool prints land
interleaved with it rather than in the application's logs, attributable to
no particular turn or tool call.
fix: >
Remove the print(). For operator diagnostics, emit through a module logger
(logging.getLogger(__name__).info(...)) so the record carries its module
and level and lands in the application's log sink; where the tool may be
served over a stdio transport, make sure that handler writes to stderr,
which the transport leaves alone. If the information needs to reach the
model, include it in the tool's return value instead.
58 changes: 58 additions & 0 deletions claude_sdk/tool_definition.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -183,3 +183,61 @@ rules:
Expand the description to at least a full sentence covering inputs,
outputs, and the situation in which this tool should be used over
alternatives.

- id: CSDK-021
title: TypeScript Claude SDK tool description is a placeholder
severity: low
confidence: 0.85
language: typescript
applies_to:
- claude_sdk_tool
scope: tool
match:
has_description_text:
- todo
- tbd
- fixme
- placeholder
- no description
- does stuff
explanation: >
The tool sets a description, so it passes CSDK-014, but the string is a
placeholder rather than real content. That leaves the tool in exactly the
state CSDK-014 exists to prevent: the TypeScript SDK has no docstring
fallback, so this argument is the entire account of the tool in the
prompt, and "TODO: describe this tool" tells the model nothing the tool
name did not. Nothing reports it — the SDK does not warn, the Zod schema
still validates, and the only symptom is a tool the model calls at the
wrong moment or never reaches for at all.
fix: >
Replace the placeholder with a real description covering what the tool
does, what it returns, and when the model should call it rather than a
neighboring tool. The SDK passes the string to the model verbatim, so
write it for the model rather than a human maintainer.

- id: CSDK-022
title: TypeScript Claude SDK tool description is too short to guide model selection
severity: low
confidence: 0.8
language: typescript
applies_to:
- claude_sdk_tool
scope: tool
match:
all:
- has_docstring: true
- description_length_lt: 40
explanation: >
A description under 40 characters is rarely enough to convey what a tool
does, what it returns, and when to call it rather than a similarly named
neighbor. With no docstring fallback in the TypeScript SDK, a stub like
"Gets data." is the whole prompt-side account of the tool, so scope and
preconditions are left to guesswork. The Zod input schema does not close
the gap: it constrains the shape of the arguments once the model has
decided to call this tool, and says nothing about whether calling it was
the right move.
fix: >
Expand the description to at least a full sentence covering inputs,
outputs, and the situation in which this tool should be used over the
alternatives. Where two tools are easy to confuse, say in each which one
the other case belongs to.
Loading
Loading