Skip to content

test(rules): mirror and cover ADK-115, ADK-116 - #166

Closed
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/adk-description-quality
Closed

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/adk-description-quality

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Engine half of a coordinated pair. Rules half: trustabl/agent-reliability-rules#89, on a branch of the same name, so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do.

What the pair adds

Ports the CSDK-017/018 description-quality pair to the Google ADK. ADK-001 only checks that a docstring exists, so """TODO: describe this tool.""" and """Gets data.""" pass today. Two ADK-specific points:

  • The docstring does double duty — the ADK parses it for the per-parameter descriptions accompanying the generated schema, so a placeholder strips argument-level guidance too.
  • In an agent tree, mis-selection doesn't stay local — a tool picked wrongly in one branch produces output the following agents treat as established fact.

ADK-116 also names the mitigation that isn't one: ADK-102/107's before_tool_callback can block a call it recognizes as wrong, but can't supply the judgment the description was meant to provide.

What this PR does

  1. Mirrors google_adk/tool_definition.yaml into testdata/rules-fixture/.
  2. Adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.
case expectation
ADK-115 — """TODO: describe this tool.""" fires
ADK-115 — real description silent
ADK-116 — """Gets data.""" fires
ADK-116 — full sentence silent
ADK-116 — no docstring at all silent

Five cases rather than four. ADK-116 pairs description_length_lt: 40 with has_docstring: true so an absent docstring stays ADK-001's finding rather than double-reporting. The fifth case pins that guard.

Verification

$ RULES_REPO=../trustabl-rules scripts/check-rules-sync.sh
rules fixture is in sync with production (86 files compared)

$ go vet ./internal/rules/
$ go test ./internal/rules/
ok  	github.com/trustabl/trustabl/internal/rules

Engine half of a coordinated pair with trustabl/agent-reliability-rules#89, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors google_adk/tool_definition.yaml into testdata/rules-fixture and adds
cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

Five cases rather than four. ADK-116 pairs description_length_lt with
has_docstring so an absent docstring stays ADK-001's finding instead of
double-reporting — an empty description is length 0, which is also under
the threshold — and the fifth case is what pins that guard.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI. The fixture copy is byte-identical to the production YAML in trustabl/agent-reliability-rules#89 (the pre-image and post-image blob hashes match across the two diffs), which is what check-rules-sync.sh wants, so the two halves need to land together.

The cases hold up when I walk them against the match expressions. parsePy fills Description from the docstring, and production ADK discovery does the same in internal/analysis/adk_agents.go, so none of these pass vacuously. The ADK-115 silent snippet contains none of the needles, the ADK-116 silent snippet is comfortably over 40 characters, and the fifth case pinning the absent-docstring path to ADK-001 rather than ADK-116 is the right instinct. That guard is the sort of thing a later edit quietly drops, and now it is held down by a test.

One nit, take it or leave it. The removed blank line at the end of policyRepoRuleCases is unrelated to this change and only adds conflict surface against your other open engine PRs that append to policies_test.go.

Nothing blocking here. I will merge this alongside the rules half.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants