Skip to content

test(rules): mirror and cover CSDK-019 - #157

Closed
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/claude-sdk-observability
Closed

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/claude-sdk-observability

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Engine half of a coordinated pair. Rules half: trustabl/agent-reliability-rules#83, on a branch of the same name, so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do.

What the pair adds

Claude SDK and MCP were the two mature packs with no observability rule (OAI-010 and ADK-009 cover the same print() pattern). Two Claude-SDK-specific consequences go beyond the lost log line:

  • A Python SDK tool is commonly served over an MCP stdio transport, where stdout carries the JSON-RPC frames — so the print corrupts the stream, not just the logs.
  • When the SDK is driven programmatically, the host application is already reading the SDK's own message stream, so tool prints land interleaved with it rather than in the application's logs.

The fix reflects the first point: where the tool may be served over stdio, the log handler has to write to stderr. Swapping print for a logger that still defaults to stdout doesn't fix that case.

What this PR does

  1. Mirrors claude_sdk/observability.yaml into testdata/rules-fixture/.
  2. Adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.
case expectation
print("looking up " + order_id) fires
logger.info(...) silent
pprint({...}) silent

The pprint case pins has_print_call's bare-callee behavior against a regression into substring matching.

Verification

$ RULES_REPO=../trustabl-rules scripts/check-rules-sync.sh
rules fixture is in sync with production (87 files compared)

$ go vet ./internal/rules/
$ go test ./internal/rules/
ok  	github.com/trustabl/trustabl/internal/rules	3.259s

Engine half of a coordinated pair with trustabl/agent-reliability-rules#83, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors claude_sdk/observability.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Three cases: the print, the module-logger remediation, and a pprint call
that must stay silent. The third pins has_print_call's bare-callee
behavior, so the rule cannot regress into substring matching that sweeps
in pprint and every other callee whose name contains "print".
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI. The mirror is exact: the fixture copy of claude_sdk/observability.yaml is byte for byte the file in trustabl/agent-reliability-rules#83, and sharing a branch name means each repo's sync job resolves the matching side rather than main, so the two validate together. No schema_version bump here is correct, since has_print_call already exists and is already dispatched at tool scope for OAI-010.

The three cases are well chosen. Neither silent case passes vacuously, because claude_sdk_tool is in applies_to, so Applies returns true and the detector really runs on the snippet. The pprint case pins PredHasPrintCall to its bare-callee behavior against a regression into substring matching, which is exactly the case I would have asked for.

Two small things. COVERAGE.md still enumerates the Claude Agent SDK Python tool rules as CSDK-001..009, 107, 108, so it should pick up 019 in the same change. And I left a question on trustabl/agent-reliability-rules#83 about the stdio framing in the rule's explanation, so whatever we settle there needs the identical edit in this file to keep the fixture in sync.

Nice, tight pair otherwise, and I am glad to take it with those tidied up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants