Skip to content

feat(autogen): add AG2-018, tool prints to stdout for diagnostics - #74

Open
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/autogen-observability
Open

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/autogen-observability

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Ports OAI-010 to AutoGen — the observability dimension isn't covered in any of the five newer packs.

AutoGen makes this especially deceptive, which is what the rule text leads with. The conversation itself is narrated to stdout, so a tool print lands in the middle of the transcript and reads as though it were part of the exchange — when in fact it's invisible to the agents reasoning over it and carries no marker for which round or which tool call produced it. Anyone reconstructing the run from a log sink rather than a terminal loses it entirely.

The fix points at the return value as "the only channel the agents actually read," which is the distinction that matters in a two-agent reply loop.

Verification — engine built at main:

$ trustabl rules validate .
OK: 86 rule pack(s), 207 rule(s) valid under rule schema version 14

Fire (print(f"looking up {order_id}") in a tool registered via register_for_llm/register_for_execution): AG2-018, AG2-201
Silent (logger.info("looking up %s", order_id)): AG2-201

has_print_call matches a bare print callee, so pprint and other attribute calls don't false-positive.

No new predicates, so no schema_version bump.

Ports OAI-010 to AutoGen. AutoGen makes it especially deceptive: the
conversation itself is narrated to stdout, so a tool print lands mid
transcript and reads as though it were part of the exchange, when it is
invisible to the agents reasoning over it and carries no marker for which
round or tool call produced it. Anyone reconstructing the run from a log
sink rather than a terminal loses it entirely.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI, this is a model contribution. I checked the rule against the engine at main: has_print_call exists in schema.go, predicates.go, and the evaluator, autogen_tool is a valid tool-scope applies_to, and you are right that no schema_version bump is needed since no new predicate is introduced. Severity and confidence sit exactly where the neighbors do (OAI-010 is low/0.65, ADK-009 low/0.7), and the pack id autogen_observability follows the <category>_<topic> convention. I also appreciate that the explanation is genuinely AutoGen framing rather than a copy of the OAI-010 text; the point about a print landing mid-transcript and reading as though it were part of the exchange is the right consequence to lead with. Picking AG2-018 to steer clear of the open PRs contesting AG2-013 through AG2-017 was thoughtful, and the ID is unclaimed by any other open PR. The engine half at trustabl/agent-reliability-analyzer#153 mirrors this file byte for byte, so rules-sync stays green as long as the two land together. One housekeeping note: every shipped rule needs a rationale doc in trustabl-rulebook before release; we can handle that side or pair on it, it is not on you to write. Approving, and I will coordinate the paired merge.

ivanpaghubasan pushed a commit to ivanpaghubasan/agent-reliability-analyzer that referenced this pull request Sep 22, 2026
Engine half of a coordinated pair with trustabl/agent-reliability-rules#74, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors autogen/observability.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Three cases: the print, the module-logger remediation the fix text
prescribes, and a pprint call that must stay silent. The third pins
has_print_call's bare-callee behavior, so the rule cannot regress into
substring matching that sweeps in pprint and every other callee whose
name contains "print".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants