Published skill-scanner evasions reduce malicious skills to passing grades

[kmondlane]

Description

Published, in-the-wild skill-scanner evasion techniques each reduce a skill
that grades F in plaintext to a passing grade (A/B/C) when scanned by
vettd. The scanner's chain detector works correctly on un-obfuscated payloads,
but two concrete, in-repo limitations — the per-file 8 KB read cap and the
silent exclusion of binary/undecodable files — let an attacker place the
same malicious behavior just outside the scanner-visible surface and receive a
clean grade.

This was reproduced against skill packages modeled on the techniques documented
by Trail of Bits, Palo Alto Unit 42, and recent academic work (ClawHavoc,
SkillCloak, ColluSkill). Reproduction skills and per-skill JSON were built and
scanned locally against vettd 0.9.3 / skill-scanner 0.1.4 (Windows 11
x86_64, debug build).

Observed results (same host, same build):

Skill	Technique	Grade	Criticals
control — overt cred-exfil, plaintext	(none)	F	yes ✅
control — cred+exfil+history-wipe chain	(none)	F	yes ✅
readme-formatter	malicious chain padded past 8 KB read cap	A	none ❌
pdf-organizer	payload in .pyc / .docx (never decoded)	A	none ❌
corp-npm-setup	registry redirect in "corporate policy" framing	B	none ❌
env-doctor	payload-less natural-language harvest+exfil	B	none ❌
video-transcriber	remote-staging curl | bash prerequisite lure	B	none ❌
config-backup + log-shipper	intent split across two skills (ColluSkill)	C / B	none ❌

Cleanest proof (truncation): the identical SSH-key exfil payload grades
F at 294 bytes and A at 323 KB (payload after ~4000 lines of filler).
The only variable is padding. Cause: contract/skill_scan.rs:23 caps each file
at MAX_READ_BYTES = 8192 before scan_skill ever sees the text.

Opaque-binary gap: run_skill_scanner walks .pyc/.docx/image files into
all_paths but excludes them from text_files, so their contents are never
inspected and produce no finding — an attacker-controllable place to hide a
payload that the runtime later executes.

The truncation mechanism overlaps #204 (coverage-honesty)
and the chain/typosquat gaps in #209; this issue tracks the security
false-negative those limits produce and the opaque-binary vector, with the
adversarial reproduction as evidence. Related: #241, #197.

Scope

• Skill-scanner false-negatives where a payload sits past the per-file read cap
  or inside a binary/undecodable file, yielding a passing grade
• Handling of binary / non-UTF-8 files that are currently walked but never
  inspected (surface them rather than silently ignore)
• Whether "no findings on the inspected surface" should be allowed to produce a
  clean grade when part of the package was not inspected
• Out of scope: LLM/semantic judgment of natural-language-only payloads,
  cross-skill composition analysis, and remote-resource fetching — real gaps
  surfaced by this test but distinct design efforts; capture as follow-ups
• Out of scope: the general coverage-honesty reporting already tracked in Scanner silently truncates its scan surface and still reports success #204

Acceptance Criteria

• A malicious chain padded past the current read cap no longer receives a
  passing grade (either the cap is raised/streamed for skill scanning, or an
  un-inspected remainder blocks a clean grade)
• Binary / undecodable files in a skill package are surfaced as an explicit
  "not analyzed" signal instead of silently omitted
• A skill is not gradeable as clean when part of its package went uninspected
• Regression fixtures cover the padded-payload and opaque-binary cases and
  assert a non-passing grade
• Cross-referenced with Scanner silently truncates its scan surface and still reports success #204 so the truncation mechanism is fixed once

[kmondlane]

Solution notes (post-open, per AGENTS.md — body stays WHAT-focused):

• Truncation: for skill scanning, raise or stream past MAX_READ_BYTES
  (contract/skill_scan.rs:23) rather than reading only the first 8 KB, or carry
  an explicit truncated: true per file into the contract so a downstream grade
  cannot be "clean" over an uninspected tail. Fix in concert with Scanner silently truncates its scan surface and still reports success #204 so the cap
  is addressed once across the MCP/agent/skill read paths.
• Opaque binaries: in run_skill_scanner, emit an explicit "not analyzed"
  finding for each path in all_paths that is absent from text_files
  (binary/undecodable), so .pyc/.docx/image payloads are surfaced instead of
  silently dropped.
• Grade gating: treat "part of the package was not inspected" as a signal
  that blocks a clean grade, mirrored in the web/certification layer
  (AgenticHighway/vettd#920).

Reproduction fixtures (padded-payload, .pyc/.docx opacity, semantic,
payload-less, remote-staging, ColluSkill split) plus two plaintext controls are
ready to land as a committed regression benchmark if we want the "evasion
benchmark" the source report recommends.

Reflected in the web app: AgenticHighway/vettd#920.