Observability detection - #159
Open
ericksondelacruz wants to merge 3 commits into
Open
ericksondelacruz wants to merge 3 commits into
ericksondelacruz wants to merge 3 commits into
Conversation
added 3 commits
September 21, 2026 15:55
Mirrors testdata/rules-fixture/ in the engine, which ships the six repo-scope observability predicates these rules evaluate and advertises schema_version 18. An older binary loads the pack leniently and skips what it cannot evaluate. - nine repo-scope absence rules, one per SDK pack (OAI-203, CSDK-208, LC-202, PYD-202, VAI-201, CREW-202, ADK-202, AG2-202, MCP-202), all severity low so a demo repo or a library does not fail CI, gated on repo_observability_inspectable so none can fire on a language the observability pass never parsed, and on repo_observability_declared: false so a repo that ships a tracing dependency it never wired gets OBS-005 instead - three agent-scope outlier rules (PYD-107, LC-112, VAI-101), gated on repo_has_observability so they read as "this agent is the outlier" rather than repeating a repo-wide complaint per agent - the new observability category: OBS-001 imported but never initialized, OBS-002 sole exporter writes to the console, OBS-003 tracing captures full prompt and response content. These deliberately widen applies_to across every SDK token because their text is vendor-framed, not SDK-framed; per-SDK copies would be byte-identical. - OBS-005, at medium: an observability package is declared in a hand-edited dependency manifest but nothing in code imports it. The dependency, the lockfile and the SBOM all attest to tracing that does not exist, which is a worse position than never having tried -- the gap survives review. Reads RepoProfile.ObsDeps and ignores poetry.lock, whose transitive closure proves nobody's intent. schema_version 17 -> 18.
Their exporter-construction and content-capture detection (pyObservabilitySignals in the engine) has no TS/JS counterpart yet, so without language: python the rules silently claimed coverage of the TS-heavy SDKs listed in their applies_to (vercel_ai, and the TS variants of claude_sdk/langchain/google_adk/mcp) that does not exist. Mirrors the same fix in the engine repo's testdata/rules-fixture.
sairenchristianbuerano
approved these changes
Sep 24, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
repo_observability_inspectableso they don't fire on a language thepass never parses.
observabilityquality category: OBS-001 (imported but never initialized, medium), OBS-002 (console-only exporter, low), OBS-003 (full prompt/response capture, medium), OBS-005 (dependency declared but neverwired, medium).
language: python— their exporter-construction and content-capture detection has no TS/JS counterpart yet, so they don't claim coverage of the TS-heavy SDKs listed inapplies_to(vercel_ai, andthe TS variants of
claude_sdk/langchain/google_adk/mcp).testdata/rules-fixture/mirror viacheck-rules-sync.sh; grounded in the pairedtrustabl-rulebookPR.main(CSKILL keyword false-positive fix, CSDK bypassPermissions gap fix) with no conflicts.Test plan
check-rules-sync.shconfirms this pack matches the engine repo's fixture 1:1go test ./...exercises every rule's fire/silent case viaTestPolicyRules_AllRulesCovered