test(rules): recover the 36 rule fixtures from the deleted fork - #218
Merged
Merged
Conversation
Engine half of a coordinated pair with trustabl/agent-reliability-rules#53, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors crewai/network.yaml into testdata/rules-fixture and adds a fire and a silent case to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. The silent case applies the remediation the rule's fix text prescribes (timeout=10) rather than deleting the call, so it demonstrates the prescribed fix actually clears the finding.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#55, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors langchain/error_handling.yaml into testdata/rules-fixture and adds a fire and a silent case to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. The silent case returns the {"error": ..., "retryable": ...} shape the rule's fix text prescribes rather than deleting the raise, so it demonstrates the prescribed remediation clears the finding. Python only. A LangChain.js counterpart is not shippable yet: PredHasRaise matches the Python grammar node "raise_statement", so has_raise is structurally always false for a TypeScript tool, whose throws parse as "throw_statement". Noted in the rules PR; closing that gap would let this pack and others cover TS error contracts.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#56, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors crewai/error_handling.yaml into testdata/rules-fixture and adds a fire and a silent case to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. The silent case returns the actionable error string CREW-008's fix text prescribes — CrewAI hands the agent a string observation rather than a dict, so the remediation shape differs from the {"error": ...} payload the other packs' rules ask for, and the case reflects that rather than copying the Claude SDK shape.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#57, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors autogen/error_handling.yaml into testdata/rules-fixture and adds a fire and a silent case to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. The silent case returns the actionable string AG2-013's fix text prescribes: AutoGen posts the tool result back as the executor agent's reply, so what the assistant reasons over is a message rather than a structured payload.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#58, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors pydantic_ai/path_safety.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases, following CSDK-004's own table: the raw path reaching open(), the .resolve() remediation the fix text prescribes, and a non-pathish param that must not drag the rule in. The third guards the per-param behavior of call_uses_unnormalized_path_param.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#60, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors crewai/path_safety.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases following CSDK-004's table: the raw path reaching open(), the .resolve() remediation the fix text prescribes, and a non-pathish param that must not drag the rule in, which guards the per-param behavior of call_uses_unnormalized_path_param.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#61, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors autogen/path_safety.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases following CSDK-004's table: the raw path reaching open(), the .resolve() remediation, and a non-pathish param that must not drag the rule in, which guards the per-param behavior of call_uses_unnormalized_path_param.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#62, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors langchain/idempotency.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the mutating tool with no key, the idempotency_key remediation, and a read-only tool name that must stay silent. The third pins both halves of the match — a missing key alone is not enough, the name has to signal a side effect — so the rule cannot quietly broaden into every tool without a key.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#64, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors pydantic_ai/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Five cases rather than four. PYD-011 pairs description_length_lt with has_docstring so an absent docstring stays PYD-001's finding instead of double-reporting, and the fifth case pins that: an empty description is length 0, which is also under the threshold, so without the guard the two rules would both fire on the same tool.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#66, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors langchain/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Five cases rather than four. LC-019 pairs description_length_lt with has_docstring so an absent docstring stays LC-001's finding instead of double-reporting, and the fifth case pins that guard.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#67, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors crewai/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Five cases rather than four. CREW-011 pairs description_length_lt with has_docstring so an absent docstring stays CREW-001's finding instead of double-reporting, and the fifth case pins that guard.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#68, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors autogen/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Five cases rather than four. AG2-017 pairs description_length_lt with has_docstring so an absent docstring stays AG2-007's finding instead of double-reporting, and the fifth case pins that guard.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#70, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors vercel_ai/path_safety.yaml into testdata/rules-fixture and adds a fire and a silent case to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. The silent case applies the remediation VAI-015's fix text prescribes for the common shape — derive the name server-side rather than accepting a path from the model — instead of merely removing the write.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#71, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors pydantic_ai/observability.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the print, the module-logger remediation the fix text prescribes, and a pprint call that must stay silent. The third pins has_print_call's bare-callee behavior, so the rule cannot regress into substring matching that sweeps in pprint and every other callee whose name contains "print".
Engine half of a coordinated pair with trustabl/agent-reliability-rules#72, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors langchain/observability.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the print, the module-logger remediation the fix text prescribes, and a pprint call that must stay silent. The third pins has_print_call's bare-callee behavior, so the rule cannot regress into substring matching that sweeps in pprint and every other callee whose name contains "print".
Engine half of a coordinated pair with trustabl/agent-reliability-rules#73, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors crewai/observability.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the print, the module-logger remediation the fix text prescribes, and a pprint call that must stay silent. The third pins has_print_call's bare-callee behavior, so the rule cannot regress into substring matching that sweeps in pprint and every other callee whose name contains "print".
Engine half of a coordinated pair with trustabl/agent-reliability-rules#74, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors autogen/observability.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the print, the module-logger remediation the fix text prescribes, and a pprint call that must stay silent. The third pins has_print_call's bare-callee behavior, so the rule cannot regress into substring matching that sweeps in pprint and every other callee whose name contains "print".
Engine half of a coordinated pair with trustabl/agent-reliability-rules#82, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors mcp/observability.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the print, the module-logger remediation, and a pprint call that must stay silent. The third pins has_print_call's bare-callee behavior, so the rule cannot regress into substring matching that sweeps in pprint and every other callee whose name contains "print".
Engine half of a coordinated pair with trustabl/agent-reliability-rules#83, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors claude_sdk/observability.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the print, the module-logger remediation, and a pprint call that must stay silent. The third pins has_print_call's bare-callee behavior, so the rule cannot regress into substring matching that sweeps in pprint and every other callee whose name contains "print".
Engine half of a coordinated pair with trustabl/agent-reliability-rules#84, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors claude_sdk/network.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. CSDK-003 covers Python; the TypeScript half was missing even though this pack ships TS rules throughout. Three cases, following OAI-016's own table: the bare fetch, the AbortSignal remediation, and a fetch whose options object is present but carries no timeout — the last pinning that the predicate checks for a deadline rather than merely for an options argument.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#85, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors mcp/network.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. MCP-004 covers Python; the TypeScript half was missing even though the pack ships TS rules (MCP-011, MCP-013). Three cases, following OAI-016's table: the bare fetch, the AbortSignal remediation, and a fetch whose options object is present but carries no timeout — the last pinning that the predicate checks for a deadline rather than merely for an options argument.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#86, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors google_adk/network.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. ADK-003 covers Python; the TypeScript half was missing even though the pack ships TS rules (ADK-013, ADK-015, ADK-016, ADK-109). Three cases, following OAI-016's table: the bare fetch, the AbortSignal remediation, and a fetch whose options object is present but carries no timeout. The snippets use adk-js's options-object form (new FunctionTool({ ..., execute })) rather than the Python FunctionTool(fn) wrapper shape, which does not discover in TS — worth noting since the Python form reads as the obvious analogue.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#87, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors openai_sdk/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Five cases rather than four. OAI-026 pairs description_length_lt with has_docstring so an absent docstring stays OAI-001's finding instead of double-reporting — an empty description is length 0, which is also under the threshold — and the fifth case is what pins that guard.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#88, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors mcp/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Five cases rather than four. MCP-026 pairs description_length_lt with has_docstring so an absent docstring stays MCP-001's finding instead of double-reporting — an empty description is length 0, which is also under the threshold — and the fifth case is what pins that guard.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#89, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors google_adk/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Five cases rather than four. ADK-116 pairs description_length_lt with has_docstring so an absent docstring stays ADK-001's finding instead of double-reporting — an empty description is length 0, which is also under the threshold — and the fifth case is what pins that guard.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#90, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors claude_sdk/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. CSDK-017/018 cover Python; the TypeScript half was missing. Five cases rather than four. CSDK-022 pairs description_length_lt with has_docstring so an empty description stays CSDK-014's finding instead of double-reporting, and the fifth case pins that guard.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#91, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors mcp/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Five cases rather than four. MCP-028 pairs description_length_lt with has_docstring so an empty description stays MCP-011's finding instead of double-reporting, and the fifth case pins that guard. Note for sequencing: rules#88 (MCP-025/026, the Python pair) appends to the same file, so whichever of the two pairs lands second needs a trivial rebase in both repos — the rule blocks and the test cases are independent and the resolution is "keep both".
Engine half of a coordinated pair with trustabl/agent-reliability-rules#92, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors openai_sdk/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Five cases rather than four. OAI-028 pairs description_length_lt with has_docstring so an absent description stays OAI-022's finding instead of double-reporting, and the fifth case pins that guard. Note for sequencing: rules#87 (OAI-025/026, the Python pair) appends to the same file, so whichever of the two pairs lands second needs a trivial rebase in both repos — the rule blocks and the test cases are independent and the resolution is "keep both".
Engine half of a coordinated pair with trustabl/agent-reliability-rules#93, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors langchain/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Five cases rather than four. LC-022 pairs description_length_lt with has_docstring so an absent description stays LC-010's finding instead of double-reporting, and the fifth case pins that guard. Note for sequencing: rules#66 (LC-018/019, the Python pair) appends to the same file, so whichever of the two pairs lands second needs a trivial rebase in both repos — the rule blocks and the test cases are independent and the resolution is "keep both".
Engine half of a coordinated pair with trustabl/agent-reliability-rules#94, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors mcp/path_safety.yaml into testdata/rules-fixture and adds a fire and a silent case to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. MCP-005 covers the Python path-safety case; the TypeScript half was missing. The silent case applies the remediation the fix text prescribes for the common shape — derive the name server-side rather than accepting a path over the protocol — instead of merely removing the write.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#95, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors openai_sdk/path_safety.yaml into testdata/rules-fixture and adds a fire and a silent case to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. OAI-006 covers the Python path-safety case; the TypeScript half was missing. The silent case applies the remediation the fix text prescribes for the common shape — derive the name server-side rather than accepting a path from the model — instead of merely removing the write.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#96, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors google_adk/path_safety.yaml into testdata/rules-fixture and adds a fire and a silent case to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. ADK-004 covers the Python path-safety case; the TypeScript half was missing. Snippets use adk-js's options-object form (new FunctionTool({ ..., execute })) rather than the Python FunctionTool(fn) wrapper shape, which does not discover in TS.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#104, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors langchain/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the generic name, the verb-object remediation, and process_invoice_batch — a name that merely contains a listed word and must stay silent. The third pins that name_in matches the whole name rather than a substring, which is the plausible regression and the one that would otherwise flood well-named tools with findings.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#105, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors crewai/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the generic name, the verb-object remediation, and process_invoice_batch — a name that merely contains a listed word and must stay silent. The third pins that name_in matches the whole name rather than a substring, which is the plausible regression and the one that would otherwise flood well-named tools with findings.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#106, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors autogen/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the generic name, the verb-object remediation, and process_invoice_batch — a name that merely contains a listed word and must stay silent. The third pins that name_in matches the whole name rather than a substring, which is the plausible regression and the one that would otherwise flood well-named tools with findings.
Engine half of a coordinated pair with trustabl/agent-reliability-rules#107, on a branch of the same name so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do. Mirrors pydantic_ai/tool_definition.yaml into testdata/rules-fixture and adds cases to policyRuleCases, as TestPolicyRules_AllRulesCovered requires. Three cases: the generic name, the verb-object remediation, and process_invoice_batch — a name that merely contains a listed word and must stay silent. The third pins that name_in matches the whole name rather than a substring, which is the plausible regression and the one that would otherwise flood well-named tools with findings.
trustabl
approved these changes
Sep 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restores the 36 rule-fixture pull requests that were closed and can no longer be reopened.
Why this is one pull request
@bradAGI deleted the
bradAGI/trustablfork, which disables GitHub's Reopen button on all 36 engine pull requests. The commits survive in this repository's ownrefs/pull/<n>/head, so every one was recovered unchanged and is still authored by @bradAGI — nothing was rewritten or re-done.They are consolidated into a single branch rather than 36 because
rules-syncfails in both directions — a rule with no fixture and a fixture with no rule — so 36 separate pairs means 36 chances to leavemainred. Two pull requests merged back to back means one short window.Verified
trustabl rules validate: 275 rules valid under schema 16, up from 228go build ./...clean,gofmtcleango test ./internal/rules/: 681 policy cases pass, 0 failcheck-rules-sync.shclean between the two branchesMerge order
Merge trustabl/agent-reliability-rules#151 and this one back to back. Either half alone turns
rules-syncred.Merge #217 first — it clears conflict markers this branch inherits from
main.The 36 pairs