Skip to content

test(rules): mirror and cover VAI-015 - #149

Closed
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/vercel-ai-path-safety
Closed

bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/vercel-ai-path-safety

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Engine half of a coordinated pair. Rules half: trustabl/agent-reliability-rules#70, on a branch of the same name, so the rules-sync job resolves the matching pack rather than main. Neither half should merge alone — check-rules-sync.sh fails if they do.

What the pair adds

The Vercel AI pack had no path-safety rule (CSDK-004/012, OAI-006, ADK-004, MCP-005 all exist). VAI-015 mirrors CSDK-012, including its coarse-signal caveat stated in the explanation: it flags any filesystem write, not only unnormalized paths, because TS path-normalization analysis isn't wired yet. Confidence 0.5 to match.

What makes it worth flagging in this pack is the deployment shape — Vercel AI tools typically run inside the same server process as the request handler rather than a sandbox, so a model-steered write inherits the application's own filesystem permissions.

What this PR does

  1. Mirrors vercel_ai/path_safety.yaml into testdata/rules-fixture/.
  2. Adds a fire case and a silent case to policyRuleCases, as TestPolicyRules_AllRulesCovered requires.

The silent case applies the remediation the fix text prescribes for the common shape — derive the filename server-side rather than accepting a path from the model — instead of merely deleting the write. So it demonstrates the prescribed fix clears the finding, not just that the predicate is satisfiable.

Verification

$ RULES_REPO=../trustabl-rules scripts/check-rules-sync.sh
rules fixture is in sync with production (87 files compared)

$ go vet ./internal/rules/
$ go test ./internal/rules/
ok  	github.com/trustabl/trustabl/internal/rules

Engine half of a coordinated pair with trustabl/agent-reliability-rules#70, on a
branch of the same name so the rules-sync job resolves the matching pack
rather than main. Neither half should merge alone — check-rules-sync.sh
fails if they do.

Mirrors vercel_ai/path_safety.yaml into testdata/rules-fixture and adds a
fire and a silent case to policyRuleCases, as
TestPolicyRules_AllRulesCovered requires.

The silent case applies the remediation VAI-015's fix text prescribes for
the common shape — derive the name server-side rather than accepting a
path from the model — instead of merely removing the write.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI, this is a clean engine half. The fixture copy of vercel_ai/path_safety.yaml is identical to the production YAML in trustabl/agent-reliability-rules#70, so check-rules-sync.sh has nothing to object to, and putting both halves on the same branch name is exactly the right way to keep the sync job resolving the matching pack rather than main.

The test cases hold up on a read rather than passing vacuously. parseTSTool routes models.KindVercelAITool through DiscoverTSVercelTools, the bare ai import gate in the snippet passes, and the execute handler goes through tsHandlerFacts, which sets writes_fs on a bare writeFileSync callee. PredHasWriteCall reads that fact on the TS branch, so the fire case genuinely fires and the silent case leaves the fact unset. No new predicate, so no SupportedSchemaVersion bump, which matches what you did.

One nit, not blocking. The silent case keeps the description "Save a note to disk for later retrieval" while writing to an in-memory Map, which reads a little oddly. Related: because has_write_call is coarse, a write confined under a root would still fire, so a silent case cannot really demonstrate the prescribed fix, only the absence of a write.

VAI-015 needs a rulebook rationale doc before it ships. That part is on us and we are happy to pair on it. I am glad to take this alongside the rules half.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants