Skip to content

feat(pydantic_ai): add PYD-012, tool prints to stdout for diagnostics - #71

Open
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/pydantic-ai-observability
Open

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/pydantic-ai-observability

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Ports OAI-010 to Pydantic AI — the observability dimension isn't covered in any of the five newer packs.

The loss is larger in this SDK than in the OpenAI pack, which is what the rule text leads with. Pydantic AI emits OpenTelemetry spans for each run and tool call, so everything around this print is already correlated to a trace — the bare print is the one diagnostic that lands outside it, unattached to the run that produced it. That's a sharper argument than "it disappears from logs," and it points at a better fix: attach the detail to the current span via Logfire or the OTel API, rather than only swapping in a module logger.

The stdio-transport corruption note from OAI-010 carries over, since a Pydantic AI tool served over an MCP stdio transport has the same JSON-RPC interleaving problem.

Verification — engine built at main:

$ trustabl rules validate .
OK: 86 rule pack(s), 207 rule(s) valid under rule schema version 14

Fire (print(f"looking up {order_id}") in the tool body): PYD-012, PYD-101, PYD-201
Silent (logger.info("looking up %s", order_id)): PYD-101, PYD-201

has_print_call matches a bare print callee, so pprint and other attribute calls don't false-positive.

No new predicates, so no schema_version bump.

Ports OAI-010 to Pydantic AI. The loss is larger in this SDK than in the
OpenAI pack, and the rule text says so: Pydantic AI emits OpenTelemetry
spans for each run and tool call, so everything around a print is already
correlated to a trace and the print is the one diagnostic landing outside
it, unattached to the run that produced it. The fix names attaching the
detail to the current span (Logfire or the OTel API) alongside the module
logger.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI, this is a well-built rule and exactly how I want ports across SDK packs done. The framing is genuinely Pydantic AI's own (the OpenTelemetry span argument, and the Logfire/OTel span attachment in the fix) rather than a copy of OAI-010's text with the SDK name swapped. Schema-wise it is clean: all required fields present, pydantic_ai_tool is valid for scope: tool, has_print_call exists in the engine at schema 14 so leaving manifest.yaml untouched is correct, and severity: low with confidence: 0.65 matches OAI-010, the right calibration anchor. PYD-012 collides with no other open PR, and I noticed you deliberately left 008 through 011 to the PRs already claiming them, which saves everyone a renumber. The engine half, trustabl/agent-reliability-analyzer#150, mirrors this YAML byte for byte and adds the required fire and silent cases, so the pair satisfies the sync contract; I will merge the two together so rules-sync never sees a divergent window. One note before this ships: every rule needs a rationale doc in the trustabl-rulebook repo (threat model, citations, severity defense). That is not on you, we can handle it or pair on it. Approving.

ivanpaghubasan pushed a commit to ivanpaghubasan/agent-reliability-analyzer that referenced this pull request Sep 22, 2026
Engine half of a coordinated pair with trustabl/agent-reliability-rules#71, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors pydantic_ai/observability.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Three cases: the print, the module-logger remediation the fix text
prescribes, and a pprint call that must stay silent. The third pins
has_print_call's bare-callee behavior, so the rule cannot regress into
substring matching that sweeps in pprint and every other callee whose
name contains "print".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants