Scan and directory output overstate what was examined and what was found

[kmondlane]

Description

A hands-on usability audit of vettd 0.9.3 / skill-scanner 0.1.4 (built from source, Windows 11 x86_64, prod endpoint) surfaced a cluster of behaviors where the CLI reports more certainty than it has: it returns a clean bill of health on paths it never examined, shows 3 of 33 flagged artifacts, and runs for 76 minutes without indicating whether it is working or hung. A second cluster makes the public directory output insufficient to support an install decision.

Findings already covered by open issues are excluded and listed at the bottom.

A. The CLI reports success it hasn't earned

A1 — scan file <directory> and scan folder <file> report "clean" and exit 0 without scanning anything.

Same directory, two commands:

$ vettd scan file  <dir>            exit 0   "No AI execution artifacts detected."
$ vettd scan folder <dir>           exit 0   3 artifact(s), 1 FLAGGED risk 58

The mirror case fails the same way:

$ vettd scan folder <a single file>  exit 0  "No AI execution artifacts detected."

vettd scan repo <a non-repo dir> is tolerant and scans correctly, so the inconsistency is between subcommands. Path validation at crates/vettd-cli/src/cli.rs:1135-1150 checks only path.exists(); a directory exists, so File passes validation and the pipeline finds no single file to scan. A file/folder mix-up returns a green light on unexamined content, in the vocabulary of safety. Related to #180 (--out given a directory), but a different surface.

A2 — The overview renders only 3 risk cards regardless of how many artifacts are flagged.

vettd scan full on this host:

RISK  33 flagged · 105 review · 647 clear  (785 artifact(s))

✗ CRITICAL  skill  risk 100    ~\.claude\plugins\...\mcp-integration\SKILL.md
✗ CRITICAL  skill  risk 91     (...)
✗ CRITICAL  skill  risk 91     (...)

… and 782 more:                (path + artifact type only — no score, no severity, no reason)

30 flagged artifacts are collapsed into the "782 more" list, visually indistinguishable from the 647 clear ones. Confirmed not lite mode: no ~/.vettd/.vettd.toml exists. Same truncation occurs in scan quick ("… and 27 more"). After a 76-minute scan the tool states 33 artifacts need attention and displays three of them.

A3 — Long scans emit nothing, and cache reuse is never disclosed.

$ vettd scan full                    exit 0 · 4,574,717 ms (76.2 min)
   stdout: 0 bytes for the entire run, then 70,590 bytes at completion
   stderr: 166 bytes total, one warning emitted at the start

$ vettd scan quick --no-cache        84,579 ms · 30 artifacts
$ vettd scan quick                   10,501 ms · 30 artifacts
   occurrences of "cache" in either run's stdout/stderr: 0

A progress indicator exists but is TTY-gated (cli.rs:1160-1164), so this applies to non-interactive and agent callers. An 8× timing swing with no explanation also means a user who edits a file and re-scans cannot tell whether the result is fresh or replayed. Distinct from #211, which covers cache internals rather than user-facing disclosure.

A4 — A warning advertises a --deep flag that does not exist.

stderr: warning: scan depth capped at 5; some files may have been skipped (use --deep for a full scan)
$ vettd scan quick --deep
error: unexpected argument '--deep' found

Not listed in vettd scan quick --help. Same class as #201 (post-scan guidance printing invalid commands), on a different code path. The warning is also emitted 5 times byte-identically in a single run.

B. Directory output cannot support an install decision

B1 — --json omits the identity fields the API returns, so duplicate entries are unaddressable.

The API returns two genuinely distinct records under one slug:

API -> {'slug':'us-federal-tax-assistant','id':'cmsi19j0j01bbad07e9sin0gz','publicId':'e3b15630-…','sourceUrl':'https://github.com/calef/…'}
API -> {'slug':'us-federal-tax-assistant','id':'cmsi16jfv019yad07zd98f0gr','publicId':'074c871b-…','sourceUrl':'https://github.com/calef/…'}

CLI --json keys: slug, name, description, version, author, category, badgeStatus,
                 overallGrade, sourceType, scannerRunCount, freshness

DirectoryCard (crates/vettd-cli/src/directory.rs:54-88) does not declare id, publicId, or sourceUrl, per the documented "limited to what we actually render" policy at directory.rs:40-44. That policy is right for a terminal table and wrong for --json, whose consumers render nothing — one struct is serving as both the human render model and the public JSON schema. Since directory view accepts only a slug, the second record is unreachable from the CLI by any route.

B2 — Duplicate entries are indistinguishable, and compare cannot disambiguate them.

[C] [ok]  us-federal-tax-assistant  github  2 scanners  Use this skill when helping users prepare…
[C] [ok]  us-federal-tax-assistant  github  2 scanners  Use this skill when helping users prepare…
[A] [ok]  visa-doc-translate        github  2 scanners  Translate visa application documents (ima…
[A] [ok]  visa-doc-translate        github  2 scanners  ビザ申請書類（画像）を英語に翻訳し…

directory view <slug> silently resolves to one record with no indication a sibling exists. directory compare <slug> <slug> with the same slug twice exits 0 and renders two identical columns without noticing. The visa-doc-translate pair differs only in description language.

B3 — Grades and freshness states have no legend, and freshness vocabulary differs per command.

Search and compare render [ok] / [offline] / [?]; view renders [current] for the same concept. Grades [A]–[F] appear with no scale, no derivation, and no explanation of what is being graded, in any help text or output. Observed mapping, undocumented: 1 medium + 12 info → B, 4 medium + 14 info → C, 1 critical + 1 medium + 17 info → F.

B4 — MCP capability scope does not affect the risk score.

mcp.json: server-filesystem "/"        -> risk 20  ✓ CLEAR
mcp.json: server-filesystem "./docs"   -> risk 20  ✓ CLEAR
mcp.json: command "echo", args ["hi"]  -> risk 20  ✓ CLEAR

All three score the mcp_config type base (risk_engine.rs:79) and nothing else. Mount scope and npx -y fetch-and-execute are not represented in any weight table. Note the scanner handles real secrets correctly — an AWS-shaped key in the same file escalates to CRITICAL/45 — so this is specifically a capability-scope gap. Adjacent to #202/#207 but not covered by either.

B5 — Risk rationale is truncated without saying so.

The highest-scoring artifact found shows risk 100 with reasons dangerous_combo:shell+network+fs (+30), keyword:shell (+15) — 45 of 100 points explained. risk_engine.rs:166 truncates the reasons list to the top 2 contributions with no "top 2 of N" label, and never shows the artifact-type base; score.min(100) at :161 clamps the total, so the pre-clamp score is unknowable. The gap scales with severity and is largest where a user most needs to understand the verdict.

Minor

• Mixed path separators inside one rendered path: ~\AppData\Roaming\Code/User\globalStorage\...\settings/.
• YAML block-scalar markers leak into descriptions: directory list shows descriptions of | and >- for feishu-wiki, bigquery-basics, firebase-basics. Present in the raw API response, so platform-side, but the CLI is where users meet it.
• README Quick start line 1 is wrong: documents vettd as an "Interactive wizard"; bare vettd prints help and exits 0 in both TTY and non-TTY modes. No wizard:: call is reachable without a subcommand — the picker is at cli.rs:1122 inside the scan handler. docs/user-flows.md describes this correctly.
• README states the full scan is uncapped: README:308 says "full scan enumerates the entire filesystem with no caps"; discovery.rs:24 sets MAX_ROOT_SCAN_FILES = 500_000 and the binary warns accordingly at runtime. Runtime behavior is correct; the doc is not. Related to Scanner silently truncates its scan surface and still reports success #204/Docs, dead code, and stale tooling have drifted from the real codebase #210.

Scope

In scope:

• Path-type validation for scan file / scan folder (A1)
• Artifact rendering limits in the overview formatter (A2)
• Non-interactive progress and cache-reuse disclosure (A3)
• The --deep warning text and its duplicate emission (A4)
• DirectoryCard field coverage for --json, and slug-ambiguity handling in view / compare (B1, B2)
• Grade and freshness legends; freshness vocabulary consistency across search / view / compare (B3)
• MCP capability-scope contribution to risk score (B4)
• Risk-rationale disclosure in the formatter (B5)
• The listed doc corrections

Out of scope:

• vettd directory download / install flows — tracked in Epic: vettd directory download command #176, Add vettd directory download command #169, Spike: end-to-end architecture for directory download #175, Spike: versioning strategy for directory download #170, Spike: scan-at-download strategy for directory download #171
• CI severity gating and exit codes — tracked in Add CI severity gating (--fail-on, diff-aware --fail-on-new, GitHub annotations/summary) #215
• Post-scan guidance printing invalid commands, and --json / --stdout redundancy — tracked in Machine-readable output flags are inconsistent, and post-scan guidance prints invalid commands #201
• Scan-surface read truncation (8 KB caps, MAX_FILES) — tracked in Scanner silently truncates its scan surface and still reports success #204
• Scoring/capability table consolidation and deep_keyword — tracked in Consolidate the hand-synced scoring/capability tables and fix the dead deep_keyword path #202, Custom detection rules cannot affect risk scores despite what the docs promise #207
• Cache internals and unbounded DB growth — tracked in Incremental-refresh cache path is effectively inert and the SQLite cache grows unbounded #211
• False-positive grading of benign files — tracked in Benign agent/skill files grade Critical / "Untrusted" from single lines #197
• Install script / packaging — tracked in Add quick install script for vettd-cli #174

Acceptance Criteria

• scan file given a directory, and scan folder given a file, fail with a message naming the correct subcommand and exit non-zero
• A scan reporting N flagged artifacts displays all N; truncation applies only to artifacts below the flagged threshold
• Non-interactive scans emit periodic progress to stderr, or the absence of progress in non-interactive mode is a documented decision
• Scan output states when results were served from cache and how to force a fresh scan
• The depth-cap warning names a flag that exists, and is emitted once per run
• directory search --json includes id, publicId, and sourceUrl; view / findings / compare accept a publicId
• directory view on an ambiguous slug indicates that more than one record matched; compare warns when given the same identity twice
• Grade and freshness values carry a legend reachable from CLI output or help
• search, view, and compare use one freshness vocabulary
• MCP filesystem mount scope contributes to the risk score, with a regression test asserting root-scope scores above project-scope
• Risk rationale shows the artifact-type base and labels any truncation, or shows all contributions
• README no longer describes bare vettd as an interactive wizard, and no longer claims the full scan is uncapped

---

_{Audit environment: vettd 0.9.3 / skill-scanner 0.1.4, built from source at b0845a0; Windows 11 26200 x86_64; prod endpoint https://vettd.agentichighway.ai; contract v2.4.0. All evidence above is from observed command output. Interactive/TTY paths were not tested — no PTY was available in the audit environment — and authenticated flows (inventory, --submit) were not exercised for lack of a valid API key.}

[kmondlane]

Implementation notes, kept out of the body per the repo convention that issues describe what rather than how. Ordered by user impact ÷ effort, not by severity.

Cheap, high leverage

1. A1 — path-type validation. Two conditions in the existing match at cli.rs:1135-1150. The arms already exist; they check !path.exists() and need path.is_dir() / path.is_file() alongside, with a message naming the sibling subcommand. This is the smallest change in the list and closes the only silent false-clear found.
2. A2 — overview truncation. print_overview already receives the full ScanReport; the cap is a display constant. Rendering every artifact at or above the flagged threshold and truncating only the clear ones needs no new data.
3. B1 — DirectoryCard fields. Add id, public_id, source_url to the struct at directory.rs:54-88. The #[serde(rename_all = "camelCase")] attribute already handles the wire names. Accepting a publicId in view/findings/compare is the follow-on and can land separately.
4. A4 — --deep. Correct the warning string to name vettd scan full, and hoist the emission so it fires once per run rather than per capped root.
5. B5 — risk rationale. risk_engine.rs:166 .take(2) — either raise the limit and label the list, or add the type base as an explicit first contribution so the shown terms sum to the score.
6. Doc corrections. README:308 (full-scan caps) and the Quick start wizard line. docs/user-flows.md is already correct on the latter and can be the source of truth.

Needs a decision first

7. A3 — progress and cache disclosure. Two sub-questions worth settling before implementing: whether non-interactive runs should emit periodic progress to stderr at all (an agent parsing stdout is unaffected either way, but a CI log benefits), and what the cache line should say. Something like 30 artifacts (28 cached, 2 rescanned) · --no-cache to force covers it, but the counts need to be threaded out of the cache layer. Worth coordinating with Incremental-refresh cache path is effectively inert and the SQLite cache grows unbounded #211 so the disclosure isn't built on a path that issue may change.
8. B4 — MCP capability scope. Requires deciding the weighting, not just wiring it: root/home-scope filesystem mounts vs. project-relative, and whether npx -y (fetch-and-execute of a remote package at launch) is its own signal. Suggest a regression test asserting root-scope scores strictly above project-scope, so the two can't silently converge again. Overlaps the table-consolidation work in Consolidate the hand-synced scoring/capability tables and fix the dead deep_keyword path #202 — probably wants to land after it, not before.
9. B2 — slug ambiguity. The display fix (a (2 records) marker, a warning when compare receives one identity twice) is straightforward, but the underlying duplicates are upstream data. Worth confirming whether the platform intends to dedupe at the source before investing in CLI-side collapsing.
10. B3 — legends. Mechanically trivial; the open question is what the grade legend should say, which depends on how overallGrade is derived server-side. The API returns a single letter, so the CLI can only explain what it is given.

Verification suggestion

A1, A2, B4 and B5 are all assertable in the existing test suite without network access. A1 in particular is a two-case regression test (scan file on a dir, scan folder on a file) that would have caught the original defect.

Not reproduced here

Interactive/TTY behavior was untested — no PTY was available in the audit environment — so the progress indicator at cli.rs:1160-1164 was read from source, not observed. If the interactive experience already covers A3 adequately, that finding narrows to non-interactive callers only.