Skip to content

test(rules): mirror and cover AG2-018 - #153

Closed
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/autogen-observability
Closed

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/autogen-observability

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Engine half of a coordinated pair. Rules half: trustabl/agent-reliability-rules#74, on a branch of the same name, so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do.

What the pair adds

AutoGen had no observability rule; OpenAI ships OAI-010 and ADK ships ADK-009 for the same print() pattern. The framing is AutoGen's own rather than a port: The conversation itself is narrated to stdout, so a tool print lands mid-transcript and reads as though it were part of the exchange — when it is invisible to the agents reasoning over it.

What this PR does

  1. Mirrors the new observability.yaml into testdata/rules-fixture/.
  2. Adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.
case expectation
print("looking up " + order_id) fires
logger.info("looking up %s", order_id) silent
pprint({...}) silent

Three cases rather than two. The pprint case pins has_print_call's bare-callee behavior — the rule must not regress into substring matching that sweeps in pprint and every other callee whose name merely contains "print". The schema calls that out explicitly as the trap has_body_text falls into, so it's worth a standing test rather than a comment.

The silent case applies the remediation the fix text prescribes (a module logger) rather than deleting the call.

Verification

$ RULES_REPO=../trustabl-rules scripts/check-rules-sync.sh
rules fixture is in sync with production (87 files compared)

$ go vet ./internal/rules/
$ go test ./internal/rules/
ok  	github.com/trustabl/trustabl/internal/rules

Engine half of a coordinated pair with trustabl/agent-reliability-rules#74, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors autogen/observability.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Three cases: the print, the module-logger remediation the fix text
prescribes, and a pprint call that must stay silent. The third pins
has_print_call's bare-callee behavior, so the rule cannot regress into
substring matching that sweeps in pprint and every other callee whose
name contains "print".
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI. I reviewed this together with trustabl/agent-reliability-rules#74, and the fixture copy is byte identical to the production pack, so the pair keeps rules-sync green as long as they merge together (and neither should land alone, as your PR body already says). The test cases are the part I want to call out. The pprint case is a genuinely useful pin, since PredHasPrintCall deliberately matches only the bare print callee and the schema comment names substring matching as the exact trap. I walked all three cases against the predicate on main: the fire case is a bare print call inside the tool body, the logging case is an attribute callee so it stays silent, and pprint is a different bare callee, so all three exercise real behavior rather than passing vacuously. Using the prescribed remediation (a module logger) as the silent input, instead of just deleting the call, is a nice touch too. The only stray bit is the unrelated blank-line removal at the end of policyRepoRuleCases, which is harmless. As noted on the rules half, AG2-018 will need a rationale doc in trustabl-rulebook before it ships; the maintainers can take that or pair on it. Approving, and I will coordinate landing the two halves together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants