Skip to content

Fix context rot and a risky skill step found by a prompt audit - #3

Merged
sergioazoc merged 3 commits into
mainfrom
fix/prompt-audit-context-rot
Sep 23, 2026
Merged

sergioazoc merged 3 commits into
mainfrom
fix/prompt-audit-context-rot

Conversation

@sergioazoc

Copy link
Copy Markdown
Owner

Summary

A prompt audit of the agent-facing surface (SKILL.md, the AGENTS.md set, REVIEW.md, the Claude Code config) against the current Claude generation. The surface is clean on the patterns that date prompts: no capitalized pressure language, no "think step by step", no pinned models, no tools or request code. What it found is context rot, two lines phrased against earlier versions, and one skill instruction that is unsafe when read literally.

  • Docs that lie today (fixed). skills/AGENTS.md said the davila7/claude-code-templates mirror was at v0.2.0 with v0.3.0 pending; #911 merged v0.3.0 on 2026-09-17 and the vendored copy matches its hash. app/AGENTS.md's boundary map left out RawSourceView, ErrorView, site-routes.ts, and site-definition.ts.
  • Dated phrasing (fixed). AGENTS.md justified the Lighthouse wording against a version the reader never saw; content/AGENTS.md said "There is no slogan", a leftover of the removed slogan, now stated as "do not add one".
  • New mechanism. doc-references.test.ts now asserts that app/AGENTS.md names every top-level component and every composable, plus a check that the assertion can fail. Run against the map on main, it flags ErrorView and RawSourceView, the rot fixed here.
  • Skill 0.3.1. The context-rot step said to verify every referenced command "still runs", which, read literally, runs deploy, publish, or migration scripts during an audit the skill declares read-only. It now checks that the script or binary is defined and runs only side-effect-free commands. The version is bumped in the SKILL.md frontmatter and both marketplace.json entries (edited by hand, under the principle-09 deny rules), with the new hash registered in skill-version.test.ts.

Test plan

  • pnpm lint, pnpm format:check, pnpm typecheck
  • pnpm generate then pnpm test: 14 files, 85 tests pass
  • claude plugin validate --strict .
  • The new completeness test fails on the main version of app/AGENTS.md (mutation check)

Follow-up after merge (outside the tree)

  • Re-sync the davila7/claude-code-templates mirror to the 0.3.1 SKILL.md.
  • SKILL.md is now 20,217 of the 20,480 bytes skill-spec.test.ts allows. The next addition needs a trim or a move to a reference file.

Generated with Claude Code

https://claude.ai/code/session_01GEVZ7EuuDgvN3ULSo6qiVF

A prompt audit of the agent-facing surface found four lines that no longer
match the repo:
- skills/AGENTS.md said the davila7 mirror was synced at v0.2.0 with v0.3.0
  pending. davila7/claude-code-templates#911 merged v0.3.0 on 2026-09-17 and the
  vendored copy matches its hash. Drop the hand-kept sync state; the sentence
  before it already says how to tell that a re-sync is pending.
- app/AGENTS.md's boundary map left out RawSourceView and ErrorView, and the
  two source-of-truth modules (site-routes.ts, site-definition.ts) the other
  docs point agents at.
- AGENTS.md justified the Lighthouse wording against a version the reader never
  saw. Drop that sentence; the rule it explains stays.
- content/AGENTS.md said "There is no slogan", a leftover of the removed
  slogan. Say what to do instead: do not add one.

Claude-Session: https://claude.ai/code/session_01GEVZ7EuuDgvN3ULSo6qiVF
doc-references only checked that the names app/AGENTS.md cites exist, so the
map could leave a component out and stay green (it did: RawSourceView and
ErrorView). Assert that every top-level component and every composable is
named, with a check that the assertion can fail. Run against the map on main,
it flags both missing components. The root AGENTS.md test map says so.

Claude-Session: https://claude.ai/code/session_01GEVZ7EuuDgvN3ULSo6qiVF
… skill as 0.3.1

The context-rot step told the agent to verify that every referenced command
"still runs". Read literally, that runs deploy, publish, or migration scripts
during an audit the skill declares read-only. Check instead that the script or
binary is defined, and run only the side-effect-free ones (lint, typecheck,
unit tests).

Bump the version to 0.3.1 in the SKILL.md frontmatter and both
marketplace.json entries, and register the new hash in skill-version.test.ts.
The davila7 mirror needs a re-sync once this ships.

Claude-Session: https://claude.ai/code/session_01GEVZ7EuuDgvN3ULSo6qiVF
@sergioazoc
sergioazoc merged commit ba42361 into main Sep 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant