Skip to content

test(rules): mirror and cover OAI-027, OAI-028 - #170

Closed
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/openai-ts-description-quality
Closed

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/openai-ts-description-quality

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Engine half of a coordinated pair. Rules half: trustabl/agent-reliability-rules#92, on a branch of the same name, so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do.

What the pair adds

The TypeScript counterpart to OAI-025/026. OAI-022 only checks that a description exists, so "TODO: describe this tool." and "Gets data." pass today. OAI-022's own text calls the description "the model's primary routing signal alongside the tool name" — which is exactly why a placeholder in that slot matters.

Two points aimed at assumed mitigations: the Zod parameters schema constrains the arguments once the model has decided to call this tool, never whether that decision was right; and with handoffs, a stub can route the conversation to the wrong agent, not just the wrong function.

What this PR does

  1. Mirrors openai_sdk/tool_definition.yaml into testdata/rules-fixture/.
  2. Adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.
case expectation
OAI-027 — "TODO: describe this tool." fires
OAI-027 — real description silent
OAI-028 — "Gets data." fires
OAI-028 — full sentence silent
OAI-028 — no description field at all silent

Five cases rather than four. OAI-028 pairs description_length_lt: 40 with has_docstring: true so an absent description stays OAI-022's finding rather than double-reporting. The fifth case pins that guard.

Sequencing note: rules#87 / this repo's #164 (the OAI-025/026 Python pair) append to the same two files, so whichever pair lands second needs a trivial rebase in both repos — resolution is "keep both." Happy to rebase on request.

Verification

$ RULES_REPO=../trustabl-rules scripts/check-rules-sync.sh
rules fixture is in sync with production (86 files compared)

$ go vet ./internal/rules/
$ go test ./internal/rules/
ok  	github.com/trustabl/trustabl/internal/rules

Engine half of a coordinated pair with trustabl/agent-reliability-rules#92, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors openai_sdk/tool_definition.yaml into testdata/rules-fixture and
adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered
requires.

Five cases rather than four. OAI-028 pairs description_length_lt with
has_docstring so an absent description stays OAI-022's finding instead of
double-reporting, and the fifth case pins that guard.

Note for sequencing: rules#87 (OAI-025/026, the Python pair) appends to
the same file, so whichever of the two pairs lands second needs a trivial
rebase in both repos — the rule blocks and the test cases are independent
and the resolution is "keep both".
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI. This is the engine half done the way I want to see it: the fixture copy of openai_sdk/tool_definition.yaml is byte identical to the production YAML in trustabl/agent-reliability-rules#92 (same pre-image and post-image blobs on both sides), so rules-sync has nothing to complain about as long as the two land together.

The test cases hold up when I walk them against the match expressions. "TODO: describe this tool." lowercases into the todo needle, "Gets data." is 10 characters against description_length_lt: 40 with a non-empty description, and both silent cases sit comfortably past the threshold. The fifth case is the one I would have asked for: with no description key at all has_docstring is false, the all: fails, and the finding stays OAI-022's rather than double reporting. Nothing vacuous in there.

Both predicates already ship at schema 14, so leaving SupportedSchemaVersion and the manifests alone is correct.

Two small things. The diff also removes a stray blank line at the end of policyRepoRuleCases, which is harmless but unrelated. And new rules need a rationale doc in trustabl-rulebook before they ship; that is on us, and I am happy to pair on it or write it.

Your rebase note against #164 is accurate, and keep-both is the right resolution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants