Skip to content

feat(crewai): add CREW-012, tool prints to stdout for diagnostics - #73

Open
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/crewai-observability
Open

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/crewai-observability

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Ports OAI-010 to CrewAI — the observability dimension isn't covered in any of the five newer packs.

CrewAI makes this worse than a lost log line, which is what the rule text leads with. A crew running with verbose=True is already writing its own narration to stdout, so a tool print gets interleaved into run commentary from several agents at once, with nothing marking which agent, task, or tool call emitted it. That's the trap: it looks like working diagnostics in a terminal and becomes unattributable the moment the crew runs anywhere else. The fix names what a module logger buys you here specifically — the record carries its own module and level and lands in the application's log sink rather than the crew's narration stream.

Verification — engine built at main:

$ trustabl rules validate .
OK: 86 rule pack(s), 207 rule(s) valid under rule schema version 14

Fire (print(f"looking up {order_id}") in the tool body): CREW-012, CREW-201
Silent (logger.info("looking up %s", order_id)): CREW-201

has_print_call matches a bare print callee, so pprint and other attribute calls don't false-positive.

No new predicates, so no schema_version bump.

Ports OAI-010 to CrewAI. CrewAI makes it worse than a lost log line: a
crew running with verbose=True is already writing its own narration to
stdout, so a tool print is interleaved into run commentary from several
agents at once with nothing marking which agent, task, or tool call
emitted it. What looks like working diagnostics in a terminal is
unattributable the moment the crew runs anywhere else.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI, this is a clean port. has_print_call and crewai_tool both exist on main, every required field is set, and severity: low with confidence: 0.65 lines up exactly with OAI-010, which is the right neighbor to match against. No other CrewAI rule keys on the same predicate, so there is no double fire, and CREW-012 is unclaimed both on main and across the other open PRs, so no renumbering is coming for you. You are also right that no schema_version bump belongs here, since the rule adds no predicate.

I particularly like that the text is CrewAI's own rather than a copy of the OpenAI framing. The verbose=True interleaving point is what justifies a separate rule instead of widening applies_to on OAI-010.

The engine half in trustabl/agent-reliability-analyzer#152 mirrors this file byte for byte and carries the fire and silent cases, so the pair lands together. One item is on us rather than on you: a shipped rule needs a rationale doc in trustabl-rulebook under docs/Policy/crewai/, carrying the threat model and the OWASP references. We can write that or pair with you on it, and it is not a blocker here. Nothing to do about badges/rules.json or the attestation files either, CI regenerates those after merge.

ivanpaghubasan pushed a commit to ivanpaghubasan/agent-reliability-analyzer that referenced this pull request Sep 22, 2026
Engine half of a coordinated pair with trustabl/agent-reliability-rules#73, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors crewai/observability.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Three cases: the print, the module-logger remediation the fix text
prescribes, and a pprint call that must stay silent. The third pins
has_print_call's bare-callee behavior, so the rule cannot regress into
substring matching that sweeps in pprint and every other callee whose
name contains "print".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants