Skip to content

feat(langchain): add LC-020, tool prints to stdout for diagnostics - #72

Open
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/langchain-observability
Open

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/langchain-observability

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Ports OAI-010 to LangChain — the observability dimension isn't covered in any of the five newer packs.

The loss is larger here than in the OpenAI pack, which is what the rule text leads with. LangChain already threads every tool start, end, and error through its callback system into LangSmith or whichever tracer is configured. So a bare print is the one diagnostic that lands outside that — detached from the run and the step that produced it, and simply absent the moment anyone debugs from a trace rather than a terminal. The fix therefore names the callback system as well as a module logger.

The stdio-transport corruption note from OAI-010 carries over, since a LangChain tool re-served over an MCP stdio transport has the same JSON-RPC interleaving problem.

Verification — engine built at main:

$ trustabl rules validate .
OK: 86 rule pack(s), 207 rule(s) valid under rule schema version 14

Fire (print(f"looking up {order_id}") in the tool body): LC-020, LC-201
Silent (logger.info("looking up %s", order_id)): LC-201

has_print_call matches a bare print callee, so pprint and other attribute calls don't false-positive.

No new predicates, so no schema_version bump.

Ports OAI-010 to LangChain. The loss is larger here than in the OpenAI
pack, and the rule text says so: LangChain already threads every tool
start, end, and error through its callback system into LangSmith or
whichever tracer is configured, so a bare print is the one diagnostic
landing outside that — detached from the run and the step that produced
it, and simply absent the moment anyone debugs from a trace rather than a
terminal.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI, this is a clean rule and a well-argued one. Checking it against main: langchain_tool is a valid applies_to at tool scope, has_print_call already exists in the engine and matches the bare print callee only, so your pprint claim holds, every required field is set, and since no new predicate is involved you are right that schema_version stays at 14. LC-020 is free on main and no other open PR claims it, which also keeps you clear of the contested LC-007 through LC-009 band. severity: low with confidence: 0.65 matches OAI-010 exactly, which is the right anchor for a rule whose false positive mode is a deliberate debug print.

The explanation is genuinely reframed around the callback system and the tracer rather than a straight port of the OpenAI text, and it names a concrete consequence, which is what I want from this pack.

This pairs with trustabl/agent-reliability-analyzer#151, which mirrors the file byte for byte and adds the coverage cases, so the two repo contract is satisfied. The remaining piece is a rulebook rationale doc for LC-020 before the rule ships. That is not on you, we can write it or pair on it.

ivanpaghubasan pushed a commit to ivanpaghubasan/agent-reliability-analyzer that referenced this pull request Sep 22, 2026
Engine half of a coordinated pair with trustabl/agent-reliability-rules#72, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors langchain/observability.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Three cases: the print, the module-logger remediation the fix text
prescribes, and a pprint call that must stay silent. The third pins
has_print_call's bare-callee behavior, so the rule cannot regress into
substring matching that sweeps in pprint and every other callee whose
name contains "print".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants