Skip to content

Commit c2e623d

Browse files
committed
feat(quality-scan): add documentation accuracy scan
Add comprehensive documentation accuracy verification to quality-scan skill: **New Scan Type: documentation** - Verifies README files against actual codebase implementation - Checks package names, command examples, API docs, file paths - Identifies outdated examples and missing documentation - Reports incorrect configuration formats and wrong version info **Pattern Detection:** - Package names not matching package.json - Commands with non-existent flags - File paths that don't exist - APIs documented but not exported - Config examples with wrong structure - Build outputs in wrong locations - Missing documentation for major features (75%+ of package) **Severity Levels:** - High: Critical errors (wrong commands, non-existent APIs) - Medium: Outdated info (wrong paths, old examples) - Low: Minor inaccuracies (version numbers, sizes) Follows Claude best practices with structured <context>, <instructions>, <pattern>, and <output_format> tags.
1 parent bbf489a commit c2e623d

2 files changed

Lines changed: 300 additions & 28 deletions

File tree

.claude/skills/quality-scan/SKILL.md

Lines changed: 30 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -6,27 +6,28 @@ description: Performs comprehensive quality scans across codebase to identify cr
66
# quality-scan
77

88
<task>
9-
Your task is to perform comprehensive quality scans across the socket-lib codebase using specialized agents to identify critical bugs, logic errors, caching issues, and workflow problems. Generate a prioritized report with actionable improvement tasks.
9+
Your task is to perform comprehensive quality scans across the socket-btm codebase using specialized agents to identify critical bugs, logic errors, caching issues, and workflow problems. Generate a prioritized report with actionable improvement tasks.
1010
</task>
1111

1212
<context>
1313
**What is Quality Scanning?**
1414
Quality scanning uses specialized AI agents to systematically analyze code for different categories of issues. Each agent type focuses on specific problem domains and reports findings with severity levels and actionable fixes.
1515

16-
**socket-lib Architecture:**
17-
Core utilities and infrastructure library for Socket.dev security tools that:
18-
- Provides shared utilities used across Socket.dev projects
19-
- Contains logger, fs helpers, process utilities
20-
- Manages build infrastructure and tooling
21-
- Provides testing utilities and helpers
22-
- Supports cross-platform compatibility
16+
**socket-btm Architecture:**
17+
This is Socket Security's binary tooling manager (BTM) that:
18+
- Builds custom Node.js binaries with Socket Security patches
19+
- Manages Node.js versions and patch synchronization
20+
- Produces minimal Node.js builds (node-smol-builder)
21+
- Processes upstream Node.js source code and applies security patches
22+
- Supports production deployments with patched Node.js
2323

2424
**Scan Types Available:**
2525
1. **critical** - Crashes, security vulnerabilities, resource leaks, data corruption
2626
2. **logic** - Algorithm errors, edge cases, type guards, off-by-one errors
2727
3. **cache** - Cache staleness, race conditions, invalidation bugs
2828
4. **workflow** - Build scripts, CI issues, cross-platform compatibility
2929
5. **security** - GitHub Actions workflow security (zizmor scanner)
30+
6. **documentation** - README accuracy, outdated docs, missing documentation
3031

3132
**Why Quality Scanning Matters:**
3233
- Catches bugs before they reach production
@@ -54,7 +55,7 @@ All agent prompts are embedded in `reference.md` with structured <context>, <ins
5455
- Proceed if codebase has uncommitted changes (warn but continue)
5556

5657
**Do ONLY:**
57-
- Run enabled scan types in priority order (critical → logic → cache → workflow → security)
58+
- Run enabled scan types in priority order (critical → logic → cache → workflow)
5859
- Generate structured findings with severity levels
5960
- Provide actionable improvement tasks with specific code changes
6061
- Report statistics and coverage metrics
@@ -103,14 +104,15 @@ Ask user which scans to run:
103104
3. **cache** - Caching issues (staleness, races, invalidation)
104105
4. **workflow** - Workflow problems (scripts, CI, git hooks)
105106
5. **security** - GitHub Actions security (template injection, cache poisoning, etc.)
107+
6. **documentation** - Documentation accuracy (README errors, outdated docs)
106108

107109
**User Interaction:**
108110
Use AskUserQuestion tool:
109111
- Question: "Which quality scans would you like to run?"
110112
- Header: "Scan Types"
111113
- multiSelect: true
112114
- Options:
113-
- "All scans (recommended)" → Run all 5 scan types
115+
- "All scans (recommended)" → Run all 4 scan types
114116
- "Critical only" → Run critical scan only
115117
- "Critical + Logic" → Run critical and logic scans
116118
- "Custom selection" → Ask user to specify which scans
@@ -124,6 +126,7 @@ Validate selected scan types exist in reference.md:
124126
- cache-scan → reference.md line ~200
125127
- workflow-scan → reference.md line ~300
126128
- security-scan → reference.md line ~400
129+
- documentation-scan → reference.md line ~810
127130

128131
If user requests non-existent scan type, report error and suggest valid types.
129132
</validation>
@@ -143,7 +146,7 @@ Task({
143146
description: "Critical bugs scan",
144147
prompt: `${CRITICAL_SCAN_PROMPT_FROM_REFERENCE_MD}
145148
146-
Focus on src/, packages/.
149+
Focus on packages/node-smol-builder/ directory and root-level scripts/.
147150
148151
Report findings in this format:
149152
- File: path/to/file.mts:lineNumber
@@ -160,7 +163,7 @@ Scan systematically and report all findings. If no issues found, state that expl
160163

161164
**For each scan:**
162165
1. Load agent prompt template from `reference.md`
163-
2. Customize for socket-lib context (focus on src/, packages/)
166+
2. Customize for socket-btm context (focus on packages/node-smol-builder/, scripts/, patches/)
164167
3. Spawn agent with Task tool using "general-purpose" subagent_type
165168
4. Capture findings from agent response
166169
5. Parse and categorize results
@@ -169,15 +172,15 @@ Scan systematically and report all findings. If no issues found, state that expl
169172
- critical (highest priority)
170173
- logic
171174
- cache
172-
- workflow
173-
- security (lowest priority)
175+
- workflow (lowest priority)
174176

175177
**Agent Prompt Sources:**
176178
- Critical scan: reference.md starting at line ~12
177179
- Logic scan: reference.md starting at line ~100
178180
- Cache scan: reference.md starting at line ~200
179181
- Workflow scan: reference.md starting at line ~300
180182
- Security scan: reference.md starting at line ~400
183+
- Documentation scan: reference.md starting at line ~810
181184

182185
<validation>
183186
For each scan completion:
@@ -197,7 +200,7 @@ Collect all findings from agents and aggregate:
197200

198201
```typescript
199202
interface Finding {
200-
file: string // "src/api/client.mts:89"
203+
file: string // "packages/node-smol-builder/src/patcher.mts:89"
201204
issue: string // "Potential null pointer access"
202205
severity: "Critical" | "High" | "Medium" | "Low"
203206
scanType: string // "critical"
@@ -238,19 +241,19 @@ Create structured quality report with all findings:
238241
# Quality Scan Report
239242

240243
**Date:** YYYY-MM-DD
241-
**Repository:** socket-lib
244+
**Repository:** socket-btm
242245
**Scans:** [list of scan types run]
243246
**Files Scanned:** N
244247
**Findings:** N critical, N high, N medium, N low
245248

246249
## Critical Issues (Priority 1) - N found
247250

248-
### src/api/client.mts:89
249-
- **Issue**: Potential null pointer access when calling API
250-
- **Pattern**: `const result = response.data.items[0].value`
251-
- **Trigger**: When API returns empty items array
252-
- **Fix**: `const items = response.data?.items ?? []; const value = items[0]?.value ?? null`
253-
- **Impact**: Crashes API client, breaks user workflows
251+
### packages/node-smol-builder/src/patcher.mts:89
252+
- **Issue**: Potential null pointer access when applying patches
253+
- **Pattern**: `const result = patches[index].apply()`
254+
- **Trigger**: When patch array has fewer elements than expected
255+
- **Fix**: `const patch = patches[index]; if (!patch) throw new Error('Patch not found'); const result = patch.apply()`
256+
- **Impact**: Crashes patch application process, build fails
254257
- **Scan**: critical
255258

256259
## High Issues (Priority 2) - N found
@@ -267,16 +270,15 @@ Create structured quality report with all findings:
267270

268271
## Scan Coverage
269272

270-
- **Critical scan**: N files analyzed in src/, packages/
271-
- **Logic scan**: N files analyzed
273+
- **Critical scan**: N files analyzed in packages/node-smol-builder/, scripts/
274+
- **Logic scan**: N files analyzed (patch logic, build scripts)
272275
- **Cache scan**: N files analyzed (if applicable)
273276
- **Workflow scan**: N files analyzed (package.json, scripts/, .github/)
274-
- **Security scan**: N workflow files analyzed (.github/workflows/)
275277

276278
## Recommendations
277279

278280
1. Address N critical issues immediately before next release
279-
2. Review N high-severity logic errors
281+
2. Review N high-severity logic errors in patch application
280282
3. Schedule N medium issues for next sprint
281283
4. Low-priority items can be addressed during refactoring
282284

@@ -285,7 +287,6 @@ Create structured quality report with all findings:
285287
[If a scan found no issues, list it here:]
286288
- Critical scan: ✓ Clean
287289
- Logic scan: ✓ Clean
288-
- Security scan: ✓ Clean
289290
```
290291

291292
**Output Report:**
@@ -356,6 +357,7 @@ See `reference.md` for detailed agent prompts with structured tags:
356357
- **cache-scan** - Invalidation, key generation, memory management, concurrency
357358
- **workflow-scan** - Scripts, package.json, git hooks, CI configuration
358359
- **security-scan** - GitHub Actions workflow security (runs zizmor scanner)
360+
- **documentation-scan** - README accuracy, outdated examples, incorrect package names, missing documentation
359361

360362
All agent prompts follow Claude best practices with <context>, <instructions>, <pattern>, <output_format>, and <quality_guidelines> tags.
361363

@@ -365,7 +367,7 @@ This skill is self-contained. No external commands needed.
365367

366368
## Context
367369

368-
This skill provides systematic code quality analysis for socket-lib by:
370+
This skill provides systematic code quality analysis for socket-btm by:
369371
- Spawning specialized agents for targeted analysis
370372
- Using Task tool to run agents autonomously
371373
- Embedding agent prompts in reference.md following best practices

0 commit comments

Comments
 (0)