From 1b23bbf3b6cb37b441da90bb51752a1c2552d928 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Thu, 3 Sep 2026 11:07:11 +0200 Subject: [PATCH 1/5] chore: sharpen doc-review and doc-write, add doc-structure audit skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reviewed the last 20 non-bot PRs (14 touched docs) against what the doc-review and doc-write skills actually ask for. doc-review found 0 Critical, 5 Improvement and ~20 Suggestion findings across those 14 PRs. Its genuinely valuable catches were all cross-file consistency checks the skill never asked for: a changelog entry documenting a nonexistent `kosli update attestation-type` (#371), a flow_template page out of sync with the schema the same PR regenerated (#296), and the one file an approvals-removal sweep missed (#345). Its weakest findings were prose polish, some already fixed at branch head. Changes: - doc-review: promote cross-file consistency, placement, redirects and anchor stability to named first-class checks; add an explicit "what not to report" bar and an 8-finding cap; stop hand-checking spelling, which the Mintlify vale-spellcheck check already enforces; drop the "what looks good" recital that the sticky comment re-renders on every push. Note that link-rot reports `skipping`, so internal links still need verifying by hand. - doc-write: add a Diátaxis-to-tab placement table. A GitHub Action reference page was authored into integrations/ and a human had to ask for the move to Reference (#305). Also add the redirects rule (3 of 14 PRs needed one), anchor stability (the CLI prints faq#boolean-flags in an error), and a table of generated paths never to hand-edit. - Both: read config/navigation.json, not docs.json, which holds only a $ref to it. - New doc-structure skill + monthly workflow: audits navigation shape and changelog coverage, which per-PR review structurally cannot see, and files issues. Its keyword check reproduces a real gap — /getting_started/attestations is the changelog's most-linked page and contained no occurrence of "summary" while three products shipped the feature (already tracked as #364, which is why the skill deduplicates against open issues before filing). - CLAUDE.md: document the automated PR checks, list the three doc skills, and drop the reference to a changelog-creator skill that does not exist. --- .claude/skills/doc-review/SKILL.md | 98 ++++++++++++++------- .claude/skills/doc-structure/SKILL.md | 117 ++++++++++++++++++++++++++ .claude/skills/doc-write/SKILL.md | 99 +++++++++++++++++----- .github/workflows/doc-review.yml | 25 ++++-- .github/workflows/doc-structure.yml | 84 ++++++++++++++++++ CLAUDE.md | 22 ++++- 6 files changed, 383 insertions(+), 62 deletions(-) create mode 100644 .claude/skills/doc-structure/SKILL.md create mode 100644 .github/workflows/doc-structure.yml diff --git a/.claude/skills/doc-review/SKILL.md b/.claude/skills/doc-review/SKILL.md index 563ec8b8..573910f1 100644 --- a/.claude/skills/doc-review/SKILL.md +++ b/.claude/skills/doc-review/SKILL.md @@ -1,49 +1,85 @@ --- name: doc-review -description: Review documentation for quality, structure, and convention compliance. Use when the user wants to review docs, audit a page or section, check if content is clear, evaluate information architecture, or assess navigation. Do NOT use when the user wants to write or fix docs — use doc-write for that. Triggers on "review the docs", "audit this page", "is this getting-started page clear", "check the implementation guide section". +description: Review documentation for quality, structure, and convention compliance. Use when the user wants to review docs, audit a page or section, check if content is clear, evaluate information architecture, or assess navigation. Do NOT use when the user wants to write or fix docs — use doc-write for that. For a periodic whole-site navigation and coverage audit that files issues, use doc-structure. Triggers on "review the docs", "audit this page", "is this getting-started page clear", "check the implementation guide section". --- -# Doc Review +# Doc review -Evaluate technical docs against the Diátaxis framework, information-architecture best practices, and the project's CLAUDE.md. +Review changed documentation against this repo's CLAUDE.md, the Diátaxis framework, and the site's information architecture. + +Most of this review's value comes from checks a human reviewer cannot do cheaply: verifying prose against generated reference pages, catching pages the change forgot to update, and questioning where a new page lives. Spend your turns there. + +## What is already checked for you + +Do not re-do this work: + +- **Spelling and Vale rules** — the `Mintlify Validation (kosli) - vale-spellcheck` check runs on every PR and enforces `styles/Kosli/AmericanSpelling.yml`. Never report a spelling finding. Never report "American spelling: pass". +- **PR title format** — the `Validate PR Title` job enforces Conventional Commits. +- **Live-docs script behavior** — the `Test live-docs scripts` job runs `pytest tests/`. + +Note that `Mintlify Validation (kosli) - link-rot` reports `skipping` on most PRs, so **internal link resolution is not reliably enforced** — keep verifying link targets yourself. ## Before reviewing -1. **Read the project's CLAUDE.md** — it defines all site-specific conventions. Use it as your compliance checklist for writing style, components, link format, and frontmatter. If no CLAUDE.md exists, read 3-4 existing pages to infer conventions. -2. **Read `docs.json`** to understand navigation structure. -3. **Determine review scope** — single page, section, or full site. If unclear, ask the user which scope they want. +1. **Read CLAUDE.md** — the compliance checklist for frontmatter, components, links, writing style, and the don'ts. +2. **Read `config/navigation.json`** for the navigation tree. Navigation lives there, not in `docs.json` — `docs.json` only holds a `$ref` to it. +3. **Read each changed file at the current branch head**, not just the diff. A finding that was already fixed in a later commit must not be reported. +4. **Determine scope** — a single page, a section, or the changed files in a PR. If unclear, ask. + +## The checks that matter most + +Run these first. They found the real defects in past reviews. + +### 1. Cross-file consistency + +A change is rarely confined to the files it touches. Grep the rest of the site for what the change contradicts. + +- **Prose vs. generated reference.** Pages under `client_reference/`, `terraform-reference/`, and `helm/` are generated. When prose names a command, flag, resource, or attribute, verify it exists in the generated page. *Precedent: a changelog entry documented `kosli update attestation-type`, a command with no reference page — it did not exist.* +- **Prose vs. generated schema.** `schemas/flow-template/v1.json` and `schemas/policy/v1.json` come from the API. When a page enumerates valid values, diff that list against the schema. *Precedent: `template-reference/flow_template.md` listed an attestation-type set the regenerated schema had already outgrown.* +- **Incomplete sweeps.** When a change removes or renames a concept, grep the whole site for the old term and list every file still using it. *Precedent: an approvals-removal PR updated `glossary.md` and `controls.md` but left `understand_kosli/how_kosli_works.md:22` still calling approvals a built-in attestation type.* + +### 2. Placement and navigation + +- Every new page must appear in `config/navigation.json`. Missing entry is **Critical**. +- **Ask whether the page is in the right tab and group**, not just whether it is listed somewhere. Apply the placement table in the `doc-write` skill. A page whose content is complete factual lookup belongs in the **Reference** tab even when it documents an integration. *Precedent: a GitHub Action reference page was first authored into `integrations/`; a human reviewer had to ask for the move to Reference. That question should come from this review.* +- Flag a new group created to hold a single page, and any page nested more than three levels below its tab. + +### 3. Redirects + +Three of the last fourteen doc PRs needed `config/redirects.json` entries. Any PR that renames, moves, or deletes a page needs one. A missing redirect for a page that was live is **Critical** — the URL is in customers' bookmarks, in CLI error output, and in the changelog. + +### 4. Anchor stability + +Headings are link targets. Before accepting a renamed heading, grep for its anchor across the repo. Some anchors are referenced from outside the docs — `faq/faq.md#boolean-flags` is emitted in CLI error messages. A renamed heading that breaks an inbound anchor is **Critical**. + +### 5. Page-level quality -## Page-level review +- **Diátaxis fit** — tutorial, how-to, reference, or explanation? Does the content match the form? Report a mismatch only when it would send a reader down the wrong path, not as a taxonomy note. +- **Frontmatter** — `title` and `description` present and accurate. +- **Links** — root-relative (`/getting_started/install`), never relative (`../install`). A relative link is **Critical**. Verify every internal target resolves to a file that exists. +- **Correctness** — commands, flags, output blocks, and screenshots that match the current product. -1. **Diátaxis classification** — Identify the doc type (tutorial, how-to, reference, explanation). Does the content match? Common issue: tutorials mixed with reference material, or how-to guides that teach instead of solving. -2. **Content quality** — Clear title? Accurate description? Complete for its doc type? Factually correct and current? -3. **Structure** — Logical flow? Appropriate depth? -4. **CLAUDE.md compliance** — Check the page against every rule and convention in the project's CLAUDE.md (frontmatter, components, links, writing style, don'ts). -5. **Link validation** — Internal links resolve to existing files? (Use Glob/Grep to verify.) +## What not to report -## Section-level review +The bar is: **would a reader be measurably better off after this change?** If not, leave it out. Specifically, never report: -1. Read all pages in the section. -2. Check navigation order — logical progression? -3. Identify gaps — missing pages for common user tasks? -4. Check Diátaxis balance — right mix of doc types? -5. Verify cross-linking between related pages. -6. Assess consistency in voice, structure, and components. +- A finding already fixed at the current branch head. +- Rewording that is a matter of taste. Grammar that is merely awkward is not a finding; grammar that is ambiguous or wrong is. +- Whitespace or column alignment inside pasted command output. +- Anything the automated checks above already cover. +- A finding you immediately talk yourself out of. If the recommendation ends in "which it already does" or "no change needed", it was never a finding. +- Praise, "what looks good" sections, and tables of checks that passed. The author knows what they wrote. Reviews are re-rendered into a sticky comment on every push, so recital costs the reader on every read. -## Site-level review +Report at most **8 findings**. If more clear the bar, report the 8 that matter most and say how many were left out. -1. Is the top-level navigation intuitive? -2. Are tutorials, how-to guides, reference, and explanation clearly separated? -3. Can new users find getting-started content quickly? Can experienced users find reference quickly? -4. Structural issues: orphaned pages, deep nesting (>3 levels), overloaded sections, missing landing pages. +## Output -## Output format +Group findings by file. For each: **Location** (`file:line`), **Issue** (one or two sentences), **Recommendation** (concrete). -Categorize findings as: -- **Critical** — Broken functionality, factual errors, missing required content. -- **Improvement** — Clarity issues, structural problems, missing best practices. -- **Suggestion** — Nice-to-have enhancements. +Categorize: -For each: **Location** (file:line or section), **Issue**, **Recommendation**. +- **Critical** — missing nav entry, relative link, broken internal link or anchor, missing redirect, factually wrong instruction. +- **Improvement** — a reader is likely to be misled, blocked, or sent to the wrong page. +- **Suggestion** — a real but minor gain. If you have no Improvements, question whether the Suggestions are worth posting at all. -End with a summary: findings by category, overall assessment, top 3 priorities. +Close with one line: counts by category, and a merge verdict. When nothing clears the bar, say exactly that in one sentence and stop — a short review is a good review. diff --git a/.claude/skills/doc-structure/SKILL.md b/.claude/skills/doc-structure/SKILL.md new file mode 100644 index 00000000..2a16327b --- /dev/null +++ b/.claude/skills/doc-structure/SKILL.md @@ -0,0 +1,117 @@ +--- +name: doc-structure +description: Periodic whole-site audit of documentation navigation, information architecture, and changelog coverage, filing GitHub issues for what it finds. Use for a scheduled or on-demand health check of the docs site as a whole — not for reviewing a single page or a pull request. Triggers on "audit the docs structure", "is the navigation still sensible", "what shipped that we never documented", "run a docs health check", "check changelog coverage". +--- + +# Doc structure audit + +Audit the whole site for navigation and coverage problems, then file one GitHub issue per finding. + +This is the counterpart to `doc-review`, which sees only the files a PR touched. Problems that no single PR causes — a group that grew past its label, a feature three products shipped that no page explains — are invisible to per-PR review and accumulate silently. This skill exists to find those. + +Run it monthly or on demand. It is read-only against the docs and write-only against the issue tracker: **never edit docs pages or navigation here.** The output is issues, so a human decides what to change. + +## Step 1 — Load current state + +```bash +gh issue list --state open --limit 100 --json number,title,labels,body +``` + +Read every open issue before auditing. The backlog already tracks known gaps, and a monthly job that re-files them is worse than one that files nothing. Keep this list in mind through every step below. + +## Step 2 — Mechanical checks + +These are cheap and either pass or fail. Run all of them. + +**Navigation integrity** — every page routed, no dangling entries: + +```bash +python3 -c " +import json +nav=json.load(open('config/navigation.json')); out=[] +def walk(o): + if isinstance(o,str): out.append(o) + elif isinstance(o,list): [walk(i) for i in o] + elif isinstance(o,dict): [walk(v) for k,v in o.items() if k in ('pages','groups','menu','tabs')] +walk(nav); open('/tmp/innav.txt','w').write('\n'.join(sorted(out))+'\n')" + +find . \( -name '*.md' -o -name '*.mdx' \) \ + | grep -vE 'node_modules|^\./\.(github|mintlify|claude)|^\./snippets|/(CLAUDE|README)\.md$' \ + | sed -E 's|^\./||; s|\.mdx?$||' | sort > /tmp/onfile.txt + +echo "--- on disk, not in nav (orphans) ---"; comm -23 /tmp/onfile.txt /tmp/innav.txt +echo "--- in nav, no such file ---"; comm -13 /tmp/onfile.txt /tmp/innav.txt +``` + +`index` is the site landing page and is correctly absent from navigation. Anything else in the first list is an orphan — the page is live but unreachable from the sidebar. Anything in the second list is a broken nav entry. + +**Shape** — walk the tree and record, for each group: depth, child count, and whether the label is sentence case. Flag: + +- Any group with exactly one child. It costs a click and a disclosure triangle and returns nothing. +- Any page more than three levels below its tab. +- Any group label in Title Case. CLAUDE.md mandates sentence case for headings, and nav labels are the most-read headings on the site. +- Any group past ~12 children with no internal grouping. +- Icon inconsistency: groups that have `icon` sitting beside sibling groups that do not. + +**Cross-reference rot** — pages nothing links to, and headings whose anchors are referenced from a page that no longer has them. + +## Step 3 — Information architecture + +Judgment, not mechanics. Read the group labels and the pages under them and ask: + +- **Does each group's label still describe its contents?** A group named for a doc type that holds a different type sends readers to the wrong place. Check the Diátaxis form of every page in a group against the group's promise. +- **Is one topic split across two homes?** Two groups that each hold half of a subject force the reader to know the org chart. Look for the same noun appearing in two top-level groups. +- **Does a top-level tab deliver what it promises?** A tab is the strongest navigational claim the site makes. A tab holding a handful of stub pages under an ambitious name over-promises. +- **Where would a reader look first?** For the five or six most common tasks, trace the path from the landing page. Count the clicks and the guesses. + +Weigh a finding by how many readers hit it. A mislabeled group at the top of the Documentation tab matters; a nesting quirk four levels into a reference section does not. + +## Step 4 — Changelog coverage + +`changelog/index.mdx` is the record of what shipped. Compare it against what the docs explain. + +**Find the pages the changelog leans on:** + +```bash +grep -o '](/[a-z_/#-]*' changelog/index.mdx | sed 's|](||; s|#.*||' | sort | uniq -c | sort -rn | head -25 +``` + +**Then, for each feature named in an entry from roughly the last quarter, check that the page the entry links to actually explains it.** Take the feature's own keyword — the flag, attribute, or noun the entry is about — and grep the linked page for it. A changelog entry that links to a page which never mentions the thing is a concrete, high-confidence gap. + +This check earns its keep. `/getting_started/attestations` is the changelog's most-linked page, and three entries across three products — a CLI `--summary` flag, a Terraform `summary` attribute, and a Platform release rendering summaries in the UI — all pointed there while the page contained no occurrence of "summary". + +Also flag: + +- A feature named in the changelog with **no** docs page mentioning it anywhere. +- A feature shipped across two or more products with docs for only one of them. Multi-product features are the ones that fall between owners. +- A **breaking change** with no corresponding guidance for readers who need to migrate. + +Skip bug fixes, internal changes, and performance work unless they change something a reader was told to do. + +Weight recent entries more heavily, and treat an entry that is several months old with still no coverage as evidence of a real gap rather than a lag. + +## Step 5 — Deduplicate + +For each candidate finding, search the open issues from step 1 for the same subject. Match on subject, not on wording — "document summary definitions on custom attestation types" and "attestations page missing --summary" are the same issue. + +If an open issue covers the finding: +- Skip it silently when the issue is adequate. +- Add a comment only when this run found something genuinely new about it — another product shipping the same feature, or a second page with the same gap. + +Prefer commenting on an existing issue over opening a near-duplicate. When unsure whether two findings are the same, they are. + +## Step 6 — File issues + +Cap each run at **8 issues**. If more findings survive, file the 8 highest-impact ones and list the rest in the run summary. A backlog nobody can work through is the same as no backlog. + +Follow the repo's conventions: + +- **Title** — `type: imperative description`, matching recent issues (`docs: document summary definitions on custom attestation types`, `automation: add kosli-dev/mcp-server to the update-changelog workflow`). +- **Labels** — `content` on nearly everything. Add `documentation` for a missing or incomplete page, `enhancement` for a structural change, `automation` for a generator or workflow fix. Add `priority: high` only for something actively misleading readers. Confirm against `gh label list` rather than assuming. +- **Body** — state the finding, the evidence that proves it (file paths, line numbers, the grep that found it, the changelog entries involved), and what a fix would look like. Someone should be able to act on the issue without re-running the audit. + +Do not assign, milestone, or set priority beyond the labels above. + +## Step 7 — Summarize + +Report: issues filed with numbers and titles, findings skipped as duplicates and which existing issue covers each, findings that cleared the bar but exceeded the cap, and one line on whether site structure improved or degraded since the last run. diff --git a/.claude/skills/doc-write/SKILL.md b/.claude/skills/doc-write/SKILL.md index 2f315da4..052b81c9 100644 --- a/.claude/skills/doc-write/SKILL.md +++ b/.claude/skills/doc-write/SKILL.md @@ -3,40 +3,93 @@ name: doc-write description: Create, write, or update documentation pages in a Mintlify-based docs site. Use when the user wants to write docs, create a new page, document a feature, add a guide or tutorial, or update existing documentation. Do NOT use for reviewing or auditing — use doc-review for that. Triggers on "write docs for X", "create a new page", "document this feature", "add a tutorial", "draft a how-to guide". --- -# Doc Write +# Doc write -Author Mintlify docs pages following the Diátaxis framework and the project's CLAUDE.md. +Author Mintlify pages for the Kosli docs site following Diátaxis and this repo's CLAUDE.md. ## Before writing -1. **Read the project's CLAUDE.md** — it defines all site-specific conventions (components, writing style, link format, frontmatter, don'ts). Follow it exactly. If no CLAUDE.md exists, read 3-4 existing pages to infer conventions. -2. **Read `docs.json`** to understand navigation structure. -3. **Read 2-3 similar pages** to match the site's voice, structure, and component usage. -4. **Search for existing content** using Grep and Glob — you may need to update rather than create. +1. **Read CLAUDE.md** — components, writing style, link format, frontmatter, and the don'ts. Follow it exactly. +2. **Read `config/navigation.json`** for the navigation tree. Navigation lives there, not in `docs.json` — `docs.json` only holds a `$ref` to it. +3. **Search for existing content** with Grep and Glob. Updating a page beats adding one; a thin new page next to an existing one splits the topic. +4. **Check open issues** — `gh issue list --state open --label content` — the gap may already be tracked, with context on what the reader needs. +5. **Read 2-3 pages in the destination group** to match voice, structure, and component usage. -## Classify the doc type (Diátaxis) +## Never hand-edit generated pages -Every page must fit one of these four types. Classify before writing: +These are generated and your edits will be overwritten on the next run: + +| Path | Source | +|---|---| +| `client_reference/` | `scripts/` — run `scripts/dev_live_docs.sh` | +| `terraform-reference/` | `.mintlify/workflows/update-terraform-reference.md` | +| `helm/` | `.github/workflows/update-cli-docs.yml` | +| `github-action-reference/` | `.mintlify/workflows/update-github-action-reference.md` | +| `schemas/` | `scripts/update_schemas.py` (from the API) | + +To change one, fix its generator. To document a command's *usage*, write a how-to that links to the generated reference rather than restating its flags — a restated flag list goes stale silently. + +## Classify the doc type | Type | Purpose | User need | Structure | |------|---------|-----------|-----------| -| **Tutorial** | Learning-oriented | "Teach me X" | Step-by-step guided learning experience with expected outcomes | -| **How-to guide** | Task-oriented | "How do I do X?" | Goal-focused steps, assumes knowledge, handles variations | -| **Reference** | Information-oriented | "What is X?" | Complete, accurate, terse technical description | -| **Explanation** | Understanding-oriented | "Why does X work this way?" | Context, background, trade-offs, alternatives | +| **Tutorial** | Learning-oriented | "Teach me X" | Guided steps with a known outcome | +| **How-to guide** | Task-oriented | "How do I do X?" | Goal-focused steps, assumes knowledge | +| **Reference** | Information-oriented | "What is X?" | Complete, accurate, terse | +| **Explanation** | Understanding-oriented | "Why does X work this way?" | Context, background, trade-offs | + +Tutorials teach through doing; how-to guides solve one problem for someone who already knows the basics. If the type is genuinely ambiguous, ask. + +## Decide where the page goes + +Classification determines placement. Getting this wrong costs a follow-up commit and a reviewer's time, so decide it before writing, not after. + +| The page is… | Tab ▸ group | +|---|---| +| A concept, or the reasoning behind a design | Documentation ▸ Understand Kosli | +| Part of the first-run sequence a new user follows in order | Documentation ▸ Getting started | +| A task an org admin performs (users, roles, auth, org-wide settings) | Documentation ▸ Administration | +| A task a user performs with Kosli | Documentation ▸ Tutorials | +| Setting up Kosli with a third-party product | Documentation ▸ Integrations | +| A specific error message or symptom | Documentation ▸ Troubleshooting | +| Complete factual lookup — CLI, API, Terraform, Helm, schema, policy | **Reference** tab | +| Rollout and adoption guidance for a team standing Kosli up | Implementation Guide | + +The Reference tab wins on content shape, not on subject. A reference page about an integration belongs in Reference — a GitHub Action reference page was once authored into `integrations/` and had to be moved in a follow-up commit. + +Note that the Documentation ▸ Tutorials group holds mostly how-to guides despite its name. Put a how-to there and follow the group's existing convention; do not create a parallel group. + +## Navigation rules + +- **Creating a page and adding it to `config/navigation.json` are one task.** A page absent from navigation does not exist on the site. +- Add it to an existing group. Only create a group when you are adding three or more sibling pages — a group wrapping a single page adds a click and gives nothing back. +- Keep pages within three levels of their tab. +- **Sentence case for group labels**, matching CLAUDE.md's heading rule: "Naming conventions", not "Naming Conventions". + +## When moving, renaming, or deleting a page + +1. Add a `config/redirects.json` entry from the old path to the new one. The old URL is in bookmarks, in CLI error output, and in changelog entries. +2. Grep the repo for the old path and update every link. +3. If a heading is being renamed, grep for its anchor first. Some anchors are referenced from outside this repo — the CLI prints `faq/faq.md#boolean-flags` in an error message. Keep the heading, or update the source that links to it. + +## Writing + +1. Classify the doc type and pick the destination from the table above. +2. Outline against the doc type. +3. Write the file. Root-relative links only (`/getting_started/install`). Frontmatter `title` and `description` are required. +4. Add the navigation entry. +5. Add redirects if anything moved. +6. Verify: does the change contradict a generated reference page, a schema, or a changelog entry? Grep and fix what it contradicts. -A common mistake is mixing tutorials with how-to guides — tutorials teach through doing, how-to guides solve specific problems. If the doc type is ambiguous, ask the user. +## Changelog -## Writing process +The changelog is for product changes, not doc changes — a new page describing an existing feature does not get an entry. When documenting a feature that *did* ship, check whether `changelog/index.mdx` already covers it, and make the entry and the page agree. Changelog entries are written by `.mintlify/workflows/update-changelog.md` from release tags; if an entry is wrong, fix the entry too. -1. **Classify** the doc type and confirm with the user if uncertain. -2. **Outline** the page structure based on the doc type. -3. **Write** the MDX file following the project's CLAUDE.md conventions. -4. **Place** the file in the correct directory and update `config/navigation.json` (or `docs.json` navigation, whichever the project uses). +Follow the existing `` format exactly and ask which `tags` value applies (`"CLI"`, `"Platform"`, `"Terraform Provider"`, `"GitHub Action"`) before writing one. -## Output +## Report -Write files directly to disk using Write/Edit tools. Then report: -- File path of the created/updated page. -- Navigation entry added (path in `config/navigation.json` or `docs.json`). -- Related pages that should cross-link to this one. +- File path created or updated. +- Navigation entry added, and which tab and group — say why that placement. +- Redirects added, if any. +- Pages that should now cross-link to this one. diff --git a/.github/workflows/doc-review.yml b/.github/workflows/doc-review.yml index d4f8bf67..92cb6a59 100644 --- a/.github/workflows/doc-review.yml +++ b/.github/workflows/doc-review.yml @@ -67,17 +67,30 @@ jobs: the `doc-review` skill (defined in `.claude/skills/doc-review/SKILL.md`) and the rules in this repo's CLAUDE.md. - Scope: only the docs files (.md, .mdx, config/navigation.json, + Scope: the docs files (.md, .mdx, config/navigation.json, docs.json) changed in this PR. Use `gh pr diff` to discover them. + You may read any file in the repo to check the change against it - + cross-file consistency is the most valuable thing you can do here. Constraints: - - Read each changed file and CLAUDE.md before commenting. - - If a new page was added, verify it is also listed in - `config/navigation.json`. Flag as Critical if missing. + - Read each changed file AT THE CURRENT BRANCH HEAD, not just the + diff. Never report something a later commit already fixed. + - Never report spelling. The `vale-spellcheck` check enforces + `styles/Kosli/AmericanSpelling.yml` on every PR. Do not restate + checks that passed, and do not include a "what looks good" + section - the sticky comment re-renders on every push. + - Prioritise the cross-file checks in the skill: prose against + generated reference pages and schemas, and incomplete sweeps + when a concept is renamed or removed. + - If a new page was added, verify it is listed in + `config/navigation.json` (Critical if missing) AND question + whether its tab and group are right for its Diátaxis type. + - If a page was moved, renamed, or deleted, verify + `config/redirects.json` has an entry. Critical if missing. - Flag relative links (e.g. `../foo`) as Critical - they must be root-relative. - - Be concise. Group findings by file. Use the doc-reviewer agent's - Critical / Improvement / Suggestion categories. + - Apply the skill's "What not to report" bar and its 8-finding + cap. If nothing clears the bar, say so in one sentence. - This review is advisory: do not request changes or approve. Note: The PR branch is already checked out in the current working diff --git a/.github/workflows/doc-structure.yml b/.github/workflows/doc-structure.yml new file mode 100644 index 00000000..5884db3c --- /dev/null +++ b/.github/workflows/doc-structure.yml @@ -0,0 +1,84 @@ +name: Doc Structure Audit + +on: + schedule: + # 09:00 UTC on the 1st of each month. + - cron: '0 9 1 * *' + workflow_dispatch: + inputs: + focus: + description: 'Optional: limit the audit to one area (e.g. "navigation", "changelog coverage", "Tutorials group")' + required: false + type: string + +# Never let two audits run at once — both would file the same issues, since +# neither can see the other's output when it deduplicates. +concurrency: + group: doc-structure + cancel-in-progress: false + +jobs: + doc-structure: + name: Docs navigation and coverage audit + runs-on: ubuntu-latest + timeout-minutes: 30 + permissions: + contents: read + issues: write + id-token: write # required: used to fetch the GitHub OIDC token + steps: + - name: Harden Runner + uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0 + with: + egress-policy: audit + + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + + - name: Run docs structure audit + uses: anthropics/claude-code-action@a874e9ecd7bb36efdad65429c6b35815f5a08f10 # v1.0.210 + env: + GH_TOKEN: ${{ github.token }} + with: + anthropic_federation_rule_id: ${{ vars.ANTHROPIC_FEDERATION_RULE_ID }} + anthropic_organization_id: ${{ vars.ANTHROPIC_ORGANIZATION_ID }} + anthropic_service_account_id: ${{ vars.ANTHROPIC_SERVICE_ACCOUNT_ID }} + # The `doc-structure` skill is defined locally in + # `.claude/skills/doc-structure/SKILL.md` and is picked up + # automatically from the checked-out repo. + # --allowedTools is a whitelist - the agent can ONLY use these tools + # plus the implicit Read/Grep/Glob set. It has no write access to the + # working tree: this audit files issues, it does not change docs. + claude_args: | + --max-turns 60 + --model claude-opus-5 + --allowedTools "Bash(python3:*),Bash(find:*),Bash(grep:*),Bash(comm:*),Bash(sed:*),Bash(sort:*),Bash(gh issue list:*),Bash(gh issue view:*),Bash(gh issue create:*),Bash(gh issue comment:*),Bash(gh label list:*)" + prompt: | + REPO: ${{ github.repository }} + + Run a full documentation structure audit of this repository. + Follow the `doc-structure` skill (defined in + `.claude/skills/doc-structure/SKILL.md`) and the rules in this + repo's CLAUDE.md. Work through its steps in order. + + ${{ github.event.inputs.focus && format('Focus this run on: {0}. Skip audit areas outside that focus.', github.event.inputs.focus) || 'Audit all areas: navigation integrity, information architecture, and changelog coverage.' }} + + Constraints: + - Read every open issue BEFORE filing anything. Most findings are + already tracked. Filing a duplicate is worse than filing nothing. + - This audit is read-only against the docs. Do not edit any page, + `config/navigation.json`, or `docs.json`. Your only writes are + GitHub issues. + - Cap the run at 8 new issues. If more findings clear the bar, + file the 8 highest-impact and list the rest in your summary. + - Every issue body must carry the evidence that proves the + finding — file paths, line numbers, the command that found it — + so it can be acted on without re-running the audit. + - If nothing clears the bar, file nothing and say so. A quiet run + is a valid result. + + Note: the repository is already checked out in the current working + directory. + + End with a summary of issues filed, findings skipped as duplicates + (naming the issue that already covers each), and findings that + exceeded the cap. diff --git a/CLAUDE.md b/CLAUDE.md index bec3e26f..0f72efc1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,6 +29,21 @@ mint a11y # Check color contrast and accessibility Requires Node.js v19+. +## Automated PR checks + +Reported by the Mintlify GitHub app, alongside this repo's own workflows: + +| Check | Enforces | Source | +|---|---|---| +| `Mintlify Validation (kosli) - vale-spellcheck` | `.vale.ini` + `styles/Kosli/AmericanSpelling.yml` | Mintlify app | +| `Mintlify Validation (kosli) - link-rot` | Link targets — but reports `skipping` on most PRs, so **don't rely on it**. Run `mint broken-links` locally (core rule 5). | Mintlify app | +| `Mintlify Deployment` | Preview build | Mintlify app | +| `Doc quality review` | The `doc-review` skill | `doc-review.yml` | +| `Validate PR Title` | Conventional Commits | `pr-quality.yml` | +| `Test live-docs scripts` | `pytest tests/` | `pr-quality.yml` | + +Because spelling is already enforced, review agents should not spend turns hand-checking it. + ## Architecture - **`docs.json`** — Central config: theme, API settings, logos. Uses `$ref` to compose from files in `config/`. @@ -38,7 +53,7 @@ Requires Node.js v19+. - **`style.css`** — Custom CSS overrides applied on top of the Mintlify theme - **`scripts/`** — Python scripts that generate "live docs" (mostly under `client_reference/`) and update navigation. See [Live docs](#live-docs) below. - **`tests/`** — pytest suite for the live-docs scripts. -- **`.github/workflows/`** — `doc-review.yml` (Claude-powered PR review), `pr-quality.yml` (link/title checks), `update-cli-docs.yml`, `update-schemas.yml`. +- **`.github/workflows/`** — `doc-review.yml` (Claude-powered PR review), `doc-structure.yml` (monthly navigation and coverage audit that files issues), `pr-quality.yml` (PR title + live-docs tests), `update-cli-docs.yml`, `update-schemas.yml`. - **`schemas/`** — Generated JSON Schema assets. See [Schemas](#schemas) below. ## Live docs @@ -118,8 +133,11 @@ description: One sentence describing the page purpose. When available, prefer skills over ad-hoc approaches: +- **Writing or updating a page** — use the `doc-write` skill (`.claude/skills/doc-write/`). +- **Reviewing a page or a PR** — use the `doc-review` skill (`.claude/skills/doc-review/`). +- **Auditing site navigation and changelog coverage** — use the `doc-structure` skill (`.claude/skills/doc-structure/`). It files issues; it never edits docs. - **PR creation** — use the `pr-creator` skill if available. -- **Changelog entries** — use the `changelog-creator` skill if available. Follow the existing `` format in `changelog/index.mdx` exactly: +- **Changelog entries** — most entries are generated by `.mintlify/workflows/update-changelog.md` from release tags. When writing one by hand, follow the existing `` format in `changelog/index.mdx` exactly: ```mdx From 08f797e253520781085d6248383c9fc71a580a42 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Thu, 3 Sep 2026 11:22:42 +0200 Subject: [PATCH 2/5] fix: stop doc-review suggesting inline fixes on generated pages doc-review runs on the automated `docs: update CLI and helm reference` PRs (allowed_bots includes github-actions), whose diffs are almost entirely machine-generated. Its behavior there is a coin flip. PR #375 got it right and said so: "they must be fixed in the CLI's command long-descriptions in kosli-dev/cli - editing them here would be silently reverted... I've deliberately not included 'Fix this' links." PR #374 got it wrong. It wrote "the durable fix is in the kosli-dev/cli generator - a hand-edit here is overwritten by the next release" and then, in the next sentence, emitted a "Fix this" link scoped to repo=kosli-dev/docs telling an agent to edit line 87 of the generated page. It also left 4 inline comments on regenerated files. PR #378 left another. Those line anchors are gone after the next release. Adds a "Generated pages" section to doc-review that distinguishes two failure modes rather than banning edits everywhere: - Deterministically regenerated (client_reference/kosli*.md, helm/k8s_reporter/*.mdx, schemas/, the kosli * nav groups, live-docs sections) - a hand-edit is deleted. No inline comments, no "Fix this" links scoped to this repo; report under "Upstream - does not block this merge" naming the upstream file. Never Critical: nothing in the PR under review can fix it. - Agent-synced (terraform-reference/, github-action-reference/) - an edit survives but drifts from upstream. Report both. - Hand-authored despite the directory (client_reference/overview.md, output_and_verbosity.md) - review normally. Regeneration only removes kosli*.md. Includes the filename-to-source mapping, verified against upstream: kosli_attest_sonar.md <- cmd/kosli/attestSonar.go, helm/k8s_reporter/karpenter.mdx <- charts/k8s-reporter/mintlify/ karpenter.md.gotmpl. Also records that `^` is the CLI's backtick convention in Go long descriptions (^--jq^, ^jq^), substituted by `kosli docs`. That makes the ^jq^ defect PR #374 found a generator escaping bug in the Accordion-title path, not a typo in the Go string - so a reviewer can now tell the two apart and point at the right fix. Same distinction applied to doc-write, whose table was too blunt, and to CLAUDE.md's Live docs section, which said "find the source script first" when for client_reference the source is another repo. --- .claude/skills/doc-review/SKILL.md | 39 ++++++++++++++++++++++++++++++ .claude/skills/doc-write/SKILL.md | 26 +++++++++++++------- .github/workflows/doc-review.yml | 15 ++++++++++++ CLAUDE.md | 21 ++++++++++++++-- 4 files changed, 90 insertions(+), 11 deletions(-) diff --git a/.claude/skills/doc-review/SKILL.md b/.claude/skills/doc-review/SKILL.md index 573910f1..e6eb3304 100644 --- a/.claude/skills/doc-review/SKILL.md +++ b/.claude/skills/doc-review/SKILL.md @@ -26,6 +26,41 @@ Note that `Mintlify Validation (kosli) - link-rot` reports `skipping` on most PR 3. **Read each changed file at the current branch head**, not just the diff. A finding that was already fixed in a later commit must not be reported. 4. **Determine scope** — a single page, a section, or the changed files in a PR. If unclear, ask. +## Generated pages: report upstream, never inline + +Most of the diff volume in this repo is machine-generated. A finding on a generated page is still worth reporting — but **reporting it as an edit to this repo is wrong**, because the next release silently reverts it and the defect ships again. + +### Two categories, two different failures + +**Deterministically regenerated — a hand-edit is destroyed.** The generator deletes and rewrites these files: + +| Page | Generated by | Fix it here | +|---|---|---| +| `client_reference/kosli*.md` | `kosli docs` from the release binary | **`kosli-dev/cli`** → `cmd/kosli/.go` | +| `helm/k8s_reporter/*.mdx` | `helm-docs` | **`kosli-dev/cli`** → `charts/k8s-reporter/mintlify/.md.gotmpl`, its `_templates.gotmpl` / `_mintlify_templates.gotmpl`, or `values.yaml` | +| `schemas/flow-template/v1.json`, `schemas/policy/v1.json` | `scripts/update_schemas.py` from the API | **`kosli-dev/server`** → the Pydantic models | +| The `kosli *` groups in `config/navigation.json` | `scripts/update-cli-nav.py` | This repo → the script | +| Live-docs sections under `client_reference/` | `scripts/add_livedocs.py` | This repo → the script and `scripts/live_docs_*_data.py` | + +The filename maps to the source: `client_reference/kosli_attest_sonar.md` ← `cmd/kosli/attestSonar.go`; `kosli_create_attestation-type.md` ← `cmd/kosli/createAttestationType.go`; `helm/k8s_reporter/karpenter.mdx` ← `charts/k8s-reporter/mintlify/karpenter.md.gotmpl`. + +**Agent-synced from an upstream source of truth — a hand-edit survives but drifts.** `terraform-reference/` (from `kosli-dev/terraform-provider-kosli`) and `github-action-reference/setup_cli_action.md` (from `kosli-dev/setup-cli-action`'s `README.md` and `action.yml`) are written by the `.mintlify/workflows/` agents. Editing them here works, but if the upstream source disagrees the next sync will undo it. Report both the page fix and the upstream mismatch. + +**Not generated, despite the directory:** `client_reference/overview.md` and `client_reference/output_and_verbosity.md` are hand-authored — the regeneration step only removes `kosli*.md`. Only the version string in `overview.md` is stamped by the workflow. Review these normally. + +### Rules + +- **Do not post an inline comment on a deterministically regenerated page.** The line it anchors to will not exist after the next release. +- **Never emit a "Fix this" link scoped to `repo=kosli-dev/docs` for a generated page.** This has happened: a review correctly wrote *"the durable fix is in the `kosli-dev/cli` generator — a hand-edit here is overwritten by the next release"* and then attached a fix link telling an agent to edit the generated file anyway. If a fix link is warranted, scope it to the upstream repo. +- Report these findings **in the top-level comment**, grouped under a single heading, each naming the upstream repo and file to change. +- Say plainly that the finding does not block the merge. The regeneration PR is a faithful copy of upstream; blocking it does not fix the source and only delays the release. + +### Tell a generator bug apart from a prose typo + +In `kosli-dev/cli`, **`^` is the convention for a backtick** inside a command's long description (`^--jq^`, `^jq^`, `^'NAME=EXPRESSION'^`), and `kosli docs` substitutes it when generating markdown. So a literal `^` surviving into a published page is a **generator escaping bug**, not a typo in the Go string — the fix is in the doc generator's rendering path, not the description. A caret that reached an `` title while the same substitution worked in body prose is exactly this bug. + +Distinguish it from genuine prose defects — British spelling, a contradictory sentence, an ambiguous clause — which do live in the Go string and are fixed by editing it. + ## The checks that matter most Run these first. They found the real defects in past reviews. @@ -68,6 +103,8 @@ The bar is: **would a reader be measurably better off after this change?** If no - Whitespace or column alignment inside pasted command output. - Anything the automated checks above already cover. - A finding you immediately talk yourself out of. If the recommendation ends in "which it already does" or "no change needed", it was never a finding. +- Sample-data churn in a regenerated page — shifted timestamps, fingerprints, commit SHAs, snapshot indices. Confirm it is only data churn and move on; do not itemise it. +- Any recommendation to hand-edit a deterministically regenerated page. If the only fix is upstream, report it upstream or not at all. - Praise, "what looks good" sections, and tables of checks that passed. The author knows what they wrote. Reviews are re-rendered into a sticky comment on every push, so recital costs the reader on every read. Report at most **8 findings**. If more clear the bar, report the 8 that matter most and say how many were left out. @@ -82,4 +119,6 @@ Categorize: - **Improvement** — a reader is likely to be misled, blocked, or sent to the wrong page. - **Suggestion** — a real but minor gain. If you have no Improvements, question whether the Suggestions are worth posting at all. +Put findings on generated pages in their own section, headed **Upstream — does not block this merge**, with the repo and file to change. Never mark one Critical: nothing in the PR under review can fix it. + Close with one line: counts by category, and a merge verdict. When nothing clears the bar, say exactly that in one sentence and stop — a short review is a good review. diff --git a/.claude/skills/doc-write/SKILL.md b/.claude/skills/doc-write/SKILL.md index 052b81c9..96a60319 100644 --- a/.claude/skills/doc-write/SKILL.md +++ b/.claude/skills/doc-write/SKILL.md @@ -15,19 +15,27 @@ Author Mintlify pages for the Kosli docs site following Diátaxis and this repo' 4. **Check open issues** — `gh issue list --state open --label content` — the gap may already be tracked, with context on what the reader needs. 5. **Read 2-3 pages in the destination group** to match voice, structure, and component usage. -## Never hand-edit generated pages +## Generated pages -These are generated and your edits will be overwritten on the next run: +**Deterministically regenerated — an edit here is deleted on the next release.** Fix the source instead: -| Path | Source | +| Path | Fix it in | |---|---| -| `client_reference/` | `scripts/` — run `scripts/dev_live_docs.sh` | -| `terraform-reference/` | `.mintlify/workflows/update-terraform-reference.md` | -| `helm/` | `.github/workflows/update-cli-docs.yml` | -| `github-action-reference/` | `.mintlify/workflows/update-github-action-reference.md` | -| `schemas/` | `scripts/update_schemas.py` (from the API) | +| `client_reference/kosli*.md` | **`kosli-dev/cli`** → `cmd/kosli/.go` (e.g. `kosli_attest_sonar.md` ← `attestSonar.go`) | +| `helm/k8s_reporter/*.mdx` | **`kosli-dev/cli`** → `charts/k8s-reporter/mintlify/.md.gotmpl` or `values.yaml` | +| `schemas/` | **`kosli-dev/server`** → the Pydantic models, then `scripts/update_schemas.py` | +| The `kosli *` groups in `config/navigation.json` | This repo → `scripts/update-cli-nav.py` | +| Live-docs sections in `client_reference/` | This repo → `scripts/add_livedocs.py`, `scripts/live_docs_*_data.py` | -To change one, fix its generator. To document a command's *usage*, write a how-to that links to the generated reference rather than restating its flags — a restated flag list goes stale silently. +Run `scripts/dev_live_docs.sh` to regenerate locally; it restores `client_reference/` on exit. + +In the CLI's Go long descriptions, **`^` means backtick** (`^--jq^`, `^jq^`) and `kosli docs` substitutes it. Write `^`, not a literal backtick, when editing those strings. + +**Agent-synced from upstream — an edit survives but will drift.** `terraform-reference/` (from `kosli-dev/terraform-provider-kosli`) and `github-action-reference/setup_cli_action.md` (from `kosli-dev/setup-cli-action`'s `README.md` and `action.yml`) are maintained by the `.mintlify/workflows/` agents. Edit the page when it is wrong, but check it against upstream first — if upstream disagrees, the next sync undoes you. + +**Hand-authored despite the directory:** `client_reference/overview.md` and `client_reference/output_and_verbosity.md`. Regeneration only removes `kosli*.md`. Edit these freely. + +To document a command's *usage*, write a how-to that links to the generated reference rather than restating its flags — a restated flag list goes stale silently. ## Classify the doc type diff --git a/.github/workflows/doc-review.yml b/.github/workflows/doc-review.yml index 92cb6a59..eabcce14 100644 --- a/.github/workflows/doc-review.yml +++ b/.github/workflows/doc-review.yml @@ -89,6 +89,21 @@ jobs: `config/redirects.json` has an entry. Critical if missing. - Flag relative links (e.g. `../foo`) as Critical - they must be root-relative. + - GENERATED PAGES: this workflow reviews the automated + `docs: update CLI and helm reference for vX.Y.Z` PRs, whose + diffs are almost entirely machine-generated. Follow the skill's + "Generated pages" section. Do NOT post inline comments on + `client_reference/kosli*.md`, `helm/k8s_reporter/*.mdx` or + `schemas/` - those lines are deleted on the next release. Do NOT + emit a "Fix this" link scoped to this repo for such a page. + Report the finding in the top-level comment under an + "Upstream - does not block this merge" heading, naming the + upstream repo and file (usually `kosli-dev/cli`, + `cmd/kosli/.go` or + `charts/k8s-reporter/mintlify/.md.gotmpl`). + - Do not itemise regenerated sample-data churn (timestamps, + fingerprints, commit SHAs). Confirm it is only churn, say so in + one line, and move on. - Apply the skill's "What not to report" bar and its 8-finding cap. If nothing clears the bar, say so in one sentence. - This review is advisory: do not request changes or approve. diff --git a/CLAUDE.md b/CLAUDE.md index 0f72efc1..c86d79b4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -60,8 +60,25 @@ Because spelling is already enforced, review agents should not spend turns hand- `client_reference/` content is partly generated by scripts in `scripts/`. Run `scripts/dev_live_docs.sh` to regenerate locally; it restores -`client_reference/` on exit. Don't hand-edit generated pages — find the -source script first. +`client_reference/` on exit. + +**Don't hand-edit generated pages** — `update-cli-docs.yml` deletes and +rewrites them on every CLI release, so an edit here is silently reverted and +the defect ships again. Fix the source: + +| Page | Fix it in | +|---|---| +| `client_reference/kosli*.md` | `kosli-dev/cli` → `cmd/kosli/.go` (`kosli_attest_sonar.md` ← `attestSonar.go`) | +| `helm/k8s_reporter/*.mdx` | `kosli-dev/cli` → `charts/k8s-reporter/mintlify/.md.gotmpl`, or `values.yaml` | +| `kosli *` nav groups in `config/navigation.json` | `scripts/update-cli-nav.py` | +| Live-docs sections | `scripts/add_livedocs.py`, `scripts/live_docs_*_data.py` | + +In the CLI's Go long descriptions, `^` means backtick (`^--jq^`) and +`kosli docs` substitutes it — so a literal `^` on a published page is a +generator escaping bug, not a typo in the description. + +`client_reference/overview.md` and `client_reference/output_and_verbosity.md` +are hand-authored; regeneration only removes `kosli*.md`. Tests for the generators live in `tests/` and run with `pytest`. From 89ed2faaf52f2469ccc6a0046a1a829e4527f3b3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Thu, 3 Sep 2026 11:26:25 +0200 Subject: [PATCH 3/5] docs: replace dated observations in doc skills with durable rules Counts and current-state claims go stale as PRs land, and a skill that cites a stale statistic teaches the wrong thing while looking authoritative. Swept all three skills and CLAUDE.md for them. - Redirects check no longer says "three of the last fourteen doc PRs needed one" - states the rule and why it gets forgotten instead. - link-rot guidance no longer asserts "reports skipping on most PRs". Now tells the reviewer to check `gh pr checks` and assume nothing validated the links unless it says otherwise, so it self-heals if Mintlify starts running it reliably. - doc-structure's changelog-coverage example moved to past tense: the attestation-summaries gap is a worked example of the check firing, not a claim about the page's current state or its current link rank. - doc-write no longer asserts the Tutorials group holds mostly how-to guides - that is exactly the kind of thing doc-structure exists to get fixed. Tells the author to read the group's pages first and follow the convention they set. The past-tense `Precedent:` items are kept as-is. A thing that happened stays having happened, and they are what make the checks concrete. --- .claude/skills/doc-review/SKILL.md | 4 ++-- .claude/skills/doc-structure/SKILL.md | 2 +- .claude/skills/doc-write/SKILL.md | 2 +- CLAUDE.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.claude/skills/doc-review/SKILL.md b/.claude/skills/doc-review/SKILL.md index e6eb3304..2157f5fb 100644 --- a/.claude/skills/doc-review/SKILL.md +++ b/.claude/skills/doc-review/SKILL.md @@ -17,7 +17,7 @@ Do not re-do this work: - **PR title format** — the `Validate PR Title` job enforces Conventional Commits. - **Live-docs script behavior** — the `Test live-docs scripts` job runs `pytest tests/`. -Note that `Mintlify Validation (kosli) - link-rot` reports `skipping` on most PRs, so **internal link resolution is not reliably enforced** — keep verifying link targets yourself. +**Link checking is the exception — verify it yourself.** `Mintlify Validation (kosli) - link-rot` is unreliable: check the PR's own check runs (`gh pr checks`), and if it reports `skipping` or is absent, nothing has validated the links. Treat that as the default and confirm every internal target resolves. ## Before reviewing @@ -81,7 +81,7 @@ A change is rarely confined to the files it touches. Grep the rest of the site f ### 3. Redirects -Three of the last fourteen doc PRs needed `config/redirects.json` entries. Any PR that renames, moves, or deletes a page needs one. A missing redirect for a page that was live is **Critical** — the URL is in customers' bookmarks, in CLI error output, and in the changelog. +Any PR that renames, moves, or deletes a page needs a `config/redirects.json` entry. This is one of the easiest things to forget, because the PR looks complete without it. A missing redirect for a page that was live is **Critical** — the URL is in customers' bookmarks, in CLI error output, and in the changelog. ### 4. Anchor stability diff --git a/.claude/skills/doc-structure/SKILL.md b/.claude/skills/doc-structure/SKILL.md index 2a16327b..b81364a9 100644 --- a/.claude/skills/doc-structure/SKILL.md +++ b/.claude/skills/doc-structure/SKILL.md @@ -78,7 +78,7 @@ grep -o '](/[a-z_/#-]*' changelog/index.mdx | sed 's|](||; s|#.*||' | sort | uni **Then, for each feature named in an entry from roughly the last quarter, check that the page the entry links to actually explains it.** Take the feature's own keyword — the flag, attribute, or noun the entry is about — and grep the linked page for it. A changelog entry that links to a page which never mentions the thing is a concrete, high-confidence gap. -This check earns its keep. `/getting_started/attestations` is the changelog's most-linked page, and three entries across three products — a CLI `--summary` flag, a Terraform `summary` attribute, and a Platform release rendering summaries in the UI — all pointed there while the page contained no occurrence of "summary". +This check has caught a real gap. Custom attestation summaries shipped across three products — a CLI `--summary` flag, a Terraform `summary` attribute, and a Platform release rendering them in the UI. All three changelog entries linked to `/getting_started/attestations` to explain the feature. That page contained no occurrence of the word "summary". Also flag: diff --git a/.claude/skills/doc-write/SKILL.md b/.claude/skills/doc-write/SKILL.md index 96a60319..0180317c 100644 --- a/.claude/skills/doc-write/SKILL.md +++ b/.claude/skills/doc-write/SKILL.md @@ -65,7 +65,7 @@ Classification determines placement. Getting this wrong costs a follow-up commit The Reference tab wins on content shape, not on subject. A reference page about an integration belongs in Reference — a GitHub Action reference page was once authored into `integrations/` and had to be moved in a follow-up commit. -Note that the Documentation ▸ Tutorials group holds mostly how-to guides despite its name. Put a how-to there and follow the group's existing convention; do not create a parallel group. +A group's label may not describe its contents — read the pages already in your chosen group before writing. Where label and contents disagree, follow the convention the existing pages set; do not create a parallel group alongside it. ## Navigation rules diff --git a/CLAUDE.md b/CLAUDE.md index c86d79b4..7057aa29 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,7 +36,7 @@ Reported by the Mintlify GitHub app, alongside this repo's own workflows: | Check | Enforces | Source | |---|---|---| | `Mintlify Validation (kosli) - vale-spellcheck` | `.vale.ini` + `styles/Kosli/AmericanSpelling.yml` | Mintlify app | -| `Mintlify Validation (kosli) - link-rot` | Link targets — but reports `skipping` on most PRs, so **don't rely on it**. Run `mint broken-links` locally (core rule 5). | Mintlify app | +| `Mintlify Validation (kosli) - link-rot` | Link targets — unreliable, frequently reports `skipping`. **Don't rely on it**; run `mint broken-links` locally (core rule 5) and check `gh pr checks` before assuming links were validated. | Mintlify app | | `Mintlify Deployment` | Preview build | Mintlify app | | `Doc quality review` | The `doc-review` skill | `doc-review.yml` | | `Validate PR Title` | Conventional Commits | `pr-quality.yml` | From 37119f00581aec009e5165077b17ebb7dd9c62b1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Thu, 3 Sep 2026 11:38:32 +0200 Subject: [PATCH 4/5] feat: extract navigation audit into scripts/audit_navigation.py The doc-structure skill carried its mechanical checks as an inline python3 -c heredoc plus a find/sed/comm pipeline. Moving them into scripts/ alongside the other docs tooling buys four things: 1. Tests. pr-quality.yml already runs `pytest tests/`, so tests/test_audit_navigation.py (22 cases) pins the behavior. One of them asserts the committed navigation has integrity, which means CLAUDE.md core rule 2 - never create a page file without adding it to navigation - is now enforced deterministically on every PR instead of resting on an LLM noticing. No new workflow job needed. 2. A narrower blast radius. doc-structure.yml can allow `Bash(python3 scripts/audit_navigation.py:*)` instead of `Bash(python3:*)`, which was arbitrary code execution, and drop find/comm entirely. 3. Determinism. The same check runs identically locally, in the cron job, and in tests, rather than being retyped by a model each run. The inline version had already shipped one such bug: `sed 's|\.mdx\?$||'` is a no-op on BSD sed, so every page looked orphaned on macOS. That case is now a regression test. 4. Reuse. `--check` for CI, `--json` for the skill, readable report for humans, `--max-nesting` / `--max-group-children` to tune. Integrity findings (orphans, dangling entries) are separated from shape findings (single-child groups, deep nesting, Title Case labels, oversized groups, inconsistent icons). Only integrity can fail a build; shape is advisory, because the script cannot tell a group that should be merged from one deliberately kept separate. The Reference > CLI Reference subtree is exempt from shape checks: it is generated by update-cli-nav.py from the CLI's own command tree, so a single-child `kosli allow` group is upstream truth rather than a defect. Without that exemption the audit reported 54 findings, 21 of them proposing to reshape generated navigation - the same mistake just fixed in doc-review. It now reports 33, all hand-maintained. Writing the tests also caught a false positive in the sentence-case check: "GitHub Actions" flagged on "Actions" because only "GitHub" was allowlisted. Added multi-word product-name handling. --- .claude/skills/doc-structure/SKILL.md | 41 ++-- .github/workflows/doc-structure.yml | 5 +- CLAUDE.md | 19 +- scripts/audit_navigation.py | 322 ++++++++++++++++++++++++++ tests/test_audit_navigation.py | 224 ++++++++++++++++++ 5 files changed, 583 insertions(+), 28 deletions(-) create mode 100644 scripts/audit_navigation.py create mode 100644 tests/test_audit_navigation.py diff --git a/.claude/skills/doc-structure/SKILL.md b/.claude/skills/doc-structure/SKILL.md index b81364a9..eafff343 100644 --- a/.claude/skills/doc-structure/SKILL.md +++ b/.claude/skills/doc-structure/SKILL.md @@ -21,39 +21,28 @@ Read every open issue before auditing. The backlog already tracks known gaps, an ## Step 2 — Mechanical checks -These are cheap and either pass or fail. Run all of them. - -**Navigation integrity** — every page routed, no dangling entries: +`scripts/audit_navigation.py` does these deterministically. Run it rather than reimplementing it — it is covered by `tests/test_audit_navigation.py`, so its behavior is pinned, and a hand-rolled shell pipeline is where the subtle bugs live. ```bash -python3 -c " -import json -nav=json.load(open('config/navigation.json')); out=[] -def walk(o): - if isinstance(o,str): out.append(o) - elif isinstance(o,list): [walk(i) for i in o] - elif isinstance(o,dict): [walk(v) for k,v in o.items() if k in ('pages','groups','menu','tabs')] -walk(nav); open('/tmp/innav.txt','w').write('\n'.join(sorted(out))+'\n')" - -find . \( -name '*.md' -o -name '*.mdx' \) \ - | grep -vE 'node_modules|^\./\.(github|mintlify|claude)|^\./snippets|/(CLAUDE|README)\.md$' \ - | sed -E 's|^\./||; s|\.mdx?$||' | sort > /tmp/onfile.txt - -echo "--- on disk, not in nav (orphans) ---"; comm -23 /tmp/onfile.txt /tmp/innav.txt -echo "--- in nav, no such file ---"; comm -13 /tmp/onfile.txt /tmp/innav.txt +python3 scripts/audit_navigation.py --json ``` -`index` is the site landing page and is correctly absent from navigation. Anything else in the first list is an orphan — the page is live but unreachable from the sidebar. Anything in the second list is a broken nav entry. +It returns three keys: + +- **`orphans`** — a page file with no navigation entry. The page is live but unreachable from the sidebar, violating CLAUDE.md rule 2. Always a finding. +- **`dangling`** — a navigation entry with no file behind it. Always a finding. +- **`shape`** — information-architecture signals, each with a `kind`, a `where` path, and a `detail`: `single-child group`, `deep nesting`, `Title Case label`, `oversized group`, `inconsistent icons`. + +Drop `--json` for a readable report; add `--check` to exit non-zero on integrity findings only. Tune with `--max-nesting` and `--max-group-children`. + +Two things the script already handles, so don't re-litigate them: -**Shape** — walk the tree and record, for each group: depth, child count, and whether the label is sentence case. Flag: +- The `Reference ▸ CLI Reference` subtree is generated by `scripts/update-cli-nav.py` and mirrors the CLI's own command tree, so it is exempt from shape checks. A single-child `kosli allow` group is upstream truth, not a defect. Never file an issue proposing to reshape it. +- `index` is the landing page, configured outside `navigation`, and is never an orphan. -- Any group with exactly one child. It costs a click and a disclosure triangle and returns nothing. -- Any page more than three levels below its tab. -- Any group label in Title Case. CLAUDE.md mandates sentence case for headings, and nav labels are the most-read headings on the site. -- Any group past ~12 children with no internal grouping. -- Icon inconsistency: groups that have `icon` sitting beside sibling groups that do not. +**Shape findings are signals, not defects.** The script cannot tell a group that should be merged from one deliberately kept separate. Weigh each by how many readers hit it, and use step 3 to decide which are worth an issue. -**Cross-reference rot** — pages nothing links to, and headings whose anchors are referenced from a page that no longer has them. +**Cross-reference rot** — not scripted; check by hand. Pages nothing links to, and anchors referenced from a page whose heading has since been renamed. ## Step 3 — Information architecture diff --git a/.github/workflows/doc-structure.yml b/.github/workflows/doc-structure.yml index 5884db3c..7e88f461 100644 --- a/.github/workflows/doc-structure.yml +++ b/.github/workflows/doc-structure.yml @@ -48,10 +48,13 @@ jobs: # --allowedTools is a whitelist - the agent can ONLY use these tools # plus the implicit Read/Grep/Glob set. It has no write access to the # working tree: this audit files issues, it does not change docs. + # The mechanical checks live in scripts/audit_navigation.py so this + # can allow that one command rather than `Bash(python3:*)`, which + # would be arbitrary code execution. claude_args: | --max-turns 60 --model claude-opus-5 - --allowedTools "Bash(python3:*),Bash(find:*),Bash(grep:*),Bash(comm:*),Bash(sed:*),Bash(sort:*),Bash(gh issue list:*),Bash(gh issue view:*),Bash(gh issue create:*),Bash(gh issue comment:*),Bash(gh label list:*)" + --allowedTools "Bash(python3 scripts/audit_navigation.py:*),Bash(grep:*),Bash(sed:*),Bash(sort:*),Bash(uniq:*),Bash(head:*),Bash(gh issue list:*),Bash(gh issue view:*),Bash(gh issue create:*),Bash(gh issue comment:*),Bash(gh label list:*)" prompt: | REPO: ${{ github.repository }} diff --git a/CLAUDE.md b/CLAUDE.md index 7057aa29..78b2b558 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,7 +40,7 @@ Reported by the Mintlify GitHub app, alongside this repo's own workflows: | `Mintlify Deployment` | Preview build | Mintlify app | | `Doc quality review` | The `doc-review` skill | `doc-review.yml` | | `Validate PR Title` | Conventional Commits | `pr-quality.yml` | -| `Test live-docs scripts` | `pytest tests/` | `pr-quality.yml` | +| `Test live-docs scripts` | `pytest tests/` — including navigation integrity, so **core rule 2 is enforced**: a page file with no `config/navigation.json` entry fails the build | `pr-quality.yml` | Because spelling is already enforced, review agents should not spend turns hand-checking it. @@ -82,6 +82,23 @@ are hand-authored; regeneration only removes `kosli*.md`. Tests for the generators live in `tests/` and run with `pytest`. +## Auditing navigation + +```bash +python scripts/audit_navigation.py # readable report +python scripts/audit_navigation.py --check # exit 1 on integrity findings +python scripts/audit_navigation.py --json # for the doc-structure skill +``` + +**Integrity** — orphaned pages (a file with no `navigation` entry, core rule 2) +and dangling entries (an entry with no file). Enforced on every PR by +`pytest tests/`. + +**Shape** — advisory information-architecture signals: single-child groups, +deep nesting, Title Case labels, oversized groups, inconsistent icons. Never +fails a build. The `Reference ▸ CLI Reference` subtree is exempt because +`update-cli-nav.py` generates it from the CLI's command tree. + ## Schemas `schemas/flow-template/v1.json` and `schemas/policy/v1.json` are the static diff --git a/scripts/audit_navigation.py b/scripts/audit_navigation.py new file mode 100644 index 00000000..3feb0744 --- /dev/null +++ b/scripts/audit_navigation.py @@ -0,0 +1,322 @@ +#!/usr/bin/env python3 +"""Audit config/navigation.json against the page files on disk. + +Two kinds of finding, deliberately separated: + +**Integrity** — objectively broken, and the only thing `--check` fails on: + +| Finding | Meaning | +|----------|----------------------------------------------------------------| +| orphan | A page file exists but no navigation entry points at it, so it | +| | is live on the site and unreachable from the sidebar. This is | +| | CLAUDE.md rule 2 ("never create a page file without also | +| | adding it to navigation"). | +| dangling | A navigation entry names a page with no file behind it. | + +**Shape** — information-architecture signals. Heuristics, reported but never +fatal: single-child groups, over-deep nesting, Title Case labels (CLAUDE.md +mandates sentence case), inconsistent icons among sibling groups, and +oversized groups. + +Usage: + python scripts/audit_navigation.py # human-readable report + python scripts/audit_navigation.py --check # exit 1 on integrity findings only + python scripts/audit_navigation.py --json # machine-readable, for the doc-structure skill +""" +import argparse +import json +import re +import sys +from pathlib import Path + +_REPO_ROOT = Path(__file__).resolve().parent.parent + +# Pages live at the repo root in topic directories. Everything here is either +# not a site page or not routed through navigation. +_EXCLUDED_DIRS = {".github", ".mintlify", ".claude", "node_modules", "snippets", "styles", "tests"} +_EXCLUDED_NAMES = {"CLAUDE.md", "README.md"} + +# The site landing page is configured outside `navigation`, so it is never an orphan. +_NOT_ROUTED = {"index"} + +# Keys whose values hold nested navigation structure. +_CONTAINER_KEYS = ("tabs", "groups", "menu", "pages") + +# Words allowed to be capitalised mid-label: proper nouns, acronyms, and +# command names. Extend this rather than loosening the sentence-case check. +_PROPER_NOUNS = { + "API", "APIs", "AWS", "Azure", "Bitbucket", "CLI", "CTRF", "Docker", "ECS", + "FAQ", "GitHub", "GitLab", "Helm", "Jira", "JSON", "JUnit", "K8S", "Karpenter", + "Kosli", "Kubernetes", "Lambda", "LaunchDarkly", "MCP", "OPA", "Rego", "S3", + "SAML", "SCIM", "SSO", "Slack", "Snyk", "Sonar", "TLS", "Terraform", +} + +# Multi-word product names, where a word that would otherwise look like Title +# Case is part of the name. Matched and removed before the word check, so +# "Actions" is allowed in "GitHub Actions" but still flagged on its own. +_PROPER_PHRASES = ( + "GitHub Actions", + "GitHub Action", + "Cloud Run", + "Evidence Vault", + "Audit Log", +) + +_MAX_NESTING_DEFAULT = 2 +_MAX_GROUP_CHILDREN_DEFAULT = 12 + +# Navigation subtrees written by a generator. Their shape mirrors an upstream +# structure (the CLI's own command tree), so a single-child `kosli allow` group +# is correct rather than a defect, and reshaping it here would be reverted on +# the next release. Integrity still applies - a dangling generated entry is a +# real bug. Keyed by the label that roots the subtree. +_GENERATED_SUBTREES = { + "CLI Reference": "scripts/update-cli-nav.py", +} + + +def load_nav(nav_file): + """Return the parsed navigation config.""" + return json.loads(Path(nav_file).read_text(encoding="utf-8")) + + +def nav_pages(nav): + """Return the set of page paths referenced anywhere in nav.""" + found = set() + + def walk(node): + if isinstance(node, str): + found.add(node) + elif isinstance(node, list): + for item in node: + walk(item) + elif isinstance(node, dict): + for key, value in node.items(): + if key in _CONTAINER_KEYS: + walk(value) + + walk(nav) + return found + + +def page_files(root): + """Return the set of extensionless page paths on disk, relative to root.""" + root = Path(root) + found = set() + for path in root.rglob("*"): + if path.suffix not in (".md", ".mdx") or not path.is_file(): + continue + rel = path.relative_to(root) + if rel.parts[0] in _EXCLUDED_DIRS or rel.name in _EXCLUDED_NAMES: + continue + found.add(rel.with_suffix("").as_posix()) + return found + + +def integrity_findings(nav, root): + """Return (orphans, dangling) as sorted lists.""" + on_disk = page_files(root) + in_nav = nav_pages(nav) + orphans = sorted(on_disk - in_nav - _NOT_ROUTED) + dangling = sorted(in_nav - on_disk) + return orphans, dangling + + +def _label(node): + """Return a node's display label, or None if it is not a labelled container.""" + for key in ("tab", "item", "group"): + if key in node: + return node[key] + return None + + +def _children(node): + for key in ("groups", "menu", "pages"): + if key in node: + return node[key] + return [] + + +def is_title_case(label): + """True if a word after the first is capitalised without being a known proper noun.""" + # Drop known multi-word product names so their internal capitals don't count, + # keeping a placeholder so the remaining words keep their position. + stripped = label + for phrase in _PROPER_PHRASES: + stripped = stripped.replace(phrase, "x" if stripped.startswith(phrase) else "x x") + + words = re.findall(r"[\w'&/-]+", stripped) + for word in words[1:]: + bare = word.strip("&/-") + if not bare or not bare[0].isupper(): + continue + if bare in _PROPER_NOUNS or bare.rstrip("s") in _PROPER_NOUNS: + continue + if bare.isupper() and len(bare) <= 4: # unlisted short acronym + continue + return True + return False + + +def walk_containers(nav): + """Yield one record per labelled container: label, path, depth, and child counts. + + `depth` counts labelled containers between the tab and this one, so a group + sitting directly under a tab has depth 1. + """ + records = [] + + def walk(node, trail): + if isinstance(node, list): + for item in node: + walk(item, trail) + return + if not isinstance(node, dict): + return + label = _label(node) + if label is None: + return + children = _children(node) + path = trail + [label] + records.append( + { + "label": label, + "path": path, + "depth": len(trail), + "children": len(children), + "pages": sum(1 for c in children if isinstance(c, str)), + "icon": bool(node.get("icon")), + # Tabs and menu items are top-level section names, not headings, + # so the sentence-case rule does not apply to them. + "is_section": "tab" in node or "item" in node, + "generated_by": next( + (_GENERATED_SUBTREES[p] for p in path if p in _GENERATED_SUBTREES), None + ), + } + ) + walk(children, path) + + walk(nav.get("tabs", []), []) + return records + + +def shape_findings(nav, max_nesting=_MAX_NESTING_DEFAULT, + max_children=_MAX_GROUP_CHILDREN_DEFAULT): + """Return a list of information-architecture findings. Never fatal.""" + records = walk_containers(nav) + findings = [] + + for record in records: + where = " > ".join(record["path"]) + + # A generated subtree mirrors an upstream structure. Reshaping it here + # would be reverted, so its shape is not this audit's business. + if record["generated_by"]: + continue + + if not record["is_section"] and record["children"] == 1: + findings.append({ + "kind": "single-child group", + "where": where, + "detail": "a group wrapping one entry costs a click and returns nothing", + }) + + if record["depth"] > max_nesting: + findings.append({ + "kind": "deep nesting", + "where": where, + "detail": f"{record['depth']} containers below its tab (limit {max_nesting})", + }) + + if not record["is_section"] and is_title_case(record["label"]): + findings.append({ + "kind": "Title Case label", + "where": where, + "detail": "CLAUDE.md mandates sentence case; nav labels are the most-read headings", + }) + + if record["children"] > max_children: + findings.append({ + "kind": "oversized group", + "where": where, + "detail": f"{record['children']} children with no internal grouping", + }) + + # Icon consistency is a property of a sibling set, not of one container. + by_parent = {} + for record in records: + if record["generated_by"]: + continue + by_parent.setdefault(" > ".join(record["path"][:-1]), []).append(record) + for parent, siblings in by_parent.items(): + if len(siblings) < 2: + continue + with_icon = [s["label"] for s in siblings if s["icon"]] + without = [s["label"] for s in siblings if not s["icon"]] + if with_icon and without: + findings.append({ + "kind": "inconsistent icons", + "where": parent or "(top level)", + "detail": f"has icons: {', '.join(with_icon)}; missing: {', '.join(without)}", + }) + + return findings + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--nav-file", default=str(_REPO_ROOT / "config" / "navigation.json"), + help="Path to navigation.json.") + parser.add_argument("--root", default=str(_REPO_ROOT), + help="Repo root to scan for page files.") + parser.add_argument("--check", action="store_true", + help="Exit 1 if there are integrity findings. Shape findings never fail.") + parser.add_argument("--json", action="store_true", dest="as_json", + help="Emit findings as JSON.") + parser.add_argument("--max-nesting", type=int, default=_MAX_NESTING_DEFAULT, + help=f"Containers allowed below a tab (default {_MAX_NESTING_DEFAULT}).") + parser.add_argument("--max-group-children", type=int, default=_MAX_GROUP_CHILDREN_DEFAULT, + help=f"Children before a group is oversized (default {_MAX_GROUP_CHILDREN_DEFAULT}).") + args = parser.parse_args() + + nav = load_nav(args.nav_file) + orphans, dangling = integrity_findings(nav, args.root) + shape = shape_findings(nav, args.max_nesting, args.max_group_children) + + if args.as_json: + json.dump({"orphans": orphans, "dangling": dangling, "shape": shape}, + sys.stdout, indent=2) + sys.stdout.write("\n") + else: + print(f"{len(nav_pages(nav))} pages in navigation, " + f"{len(page_files(args.root))} page files on disk\n") + + if orphans: + print(f"ORPHANS ({len(orphans)}) - live but unreachable from the sidebar:") + for page in orphans: + print(f" {page}") + if dangling: + print(f"\nDANGLING ({len(dangling)}) - navigation entry with no file:") + for page in dangling: + print(f" {page}") + if not orphans and not dangling: + print("integrity: ok") + + if shape: + print(f"\nSHAPE ({len(shape)}) - advisory, never fails the build:") + for finding in shape: + print(f" [{finding['kind']}] {finding['where']}") + print(f" {finding['detail']}") + + if args.check and (orphans or dangling): + print( + f"\n{len(orphans)} orphaned page(s) and {len(dangling)} dangling entry(s). " + "Every page file must be listed in navigation (CLAUDE.md rule 2).", + file=sys.stderr, + ) + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/tests/test_audit_navigation.py b/tests/test_audit_navigation.py new file mode 100644 index 00000000..c8c01c06 --- /dev/null +++ b/tests/test_audit_navigation.py @@ -0,0 +1,224 @@ +import json +import subprocess +import sys +from pathlib import Path + +import audit_navigation as nav_audit + +_SCRIPT = Path(__file__).resolve().parent.parent / "scripts" / "audit_navigation.py" + + +def _write_site(tmp_path, nav, pages): + """Build a throwaway docs tree: nav config plus page files.""" + nav_file = tmp_path / "config" / "navigation.json" + nav_file.parent.mkdir(parents=True) + nav_file.write_text(json.dumps(nav), encoding="utf-8") + for page in pages: + target = tmp_path / page + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text("---\ntitle: x\n---\n", encoding="utf-8") + return nav_file + + +# --- page discovery --------------------------------------------------------- + +def test_page_files_strips_both_extensions(tmp_path): + """Regression: a broken extension strip made every page look orphaned.""" + _write_site(tmp_path, {"tabs": []}, ["a/one.md", "a/two.mdx"]) + assert nav_audit.page_files(tmp_path) == {"a/one", "a/two"} + + +def test_page_files_skips_excluded_dirs_and_names(tmp_path): + _write_site(tmp_path, {"tabs": []}, [ + "real/page.md", + "snippets/frag.mdx", + ".claude/skills/doc-write/SKILL.md", + ".github/notes.md", + "CLAUDE.md", + "README.md", + ]) + assert nav_audit.page_files(tmp_path) == {"real/page"} + + +# --- navigation traversal --------------------------------------------------- + +def test_nav_pages_walks_tabs_groups_menu_and_nested_pages(): + nav = { + "tabs": [ + {"tab": "Docs", "groups": [ + {"group": "A", "pages": [ + "a/one", + {"group": "B", "pages": ["a/b/two"]}, + ]}, + ]}, + {"tab": "Ref", "menu": [ + {"item": "CLI", "groups": [{"group": "C", "pages": ["ref/three"]}]}, + {"item": "API", "openapi": "https://example.test/openapi.json"}, + ]}, + ] + } + assert nav_audit.nav_pages(nav) == {"a/one", "a/b/two", "ref/three"} + + +# --- integrity -------------------------------------------------------------- + +def test_orphan_is_reported(tmp_path): + nav = {"tabs": [{"tab": "Docs", "groups": [{"group": "A", "pages": ["a/one"]}]}]} + nav_file = _write_site(tmp_path, nav, ["a/one.md", "a/lonely.md"]) + orphans, dangling = nav_audit.integrity_findings(nav_audit.load_nav(nav_file), tmp_path) + assert orphans == ["a/lonely"] + assert dangling == [] + + +def test_dangling_entry_is_reported(tmp_path): + nav = {"tabs": [{"tab": "Docs", "groups": [ + {"group": "A", "pages": ["a/one", "a/ghost"]}]}]} + nav_file = _write_site(tmp_path, nav, ["a/one.md"]) + orphans, dangling = nav_audit.integrity_findings(nav_audit.load_nav(nav_file), tmp_path) + assert orphans == [] + assert dangling == ["a/ghost"] + + +def test_index_is_never_an_orphan(tmp_path): + """The landing page is configured outside `navigation`.""" + nav_file = _write_site(tmp_path, {"tabs": []}, ["index.mdx"]) + orphans, _ = nav_audit.integrity_findings(nav_audit.load_nav(nav_file), tmp_path) + assert orphans == [] + + +def test_clean_site_has_no_integrity_findings(tmp_path): + nav = {"tabs": [{"tab": "Docs", "groups": [{"group": "A", "pages": ["a/one"]}]}]} + nav_file = _write_site(tmp_path, nav, ["a/one.md", "index.mdx"]) + assert nav_audit.integrity_findings(nav_audit.load_nav(nav_file), tmp_path) == ([], []) + + +# --- sentence case ---------------------------------------------------------- + +def test_sentence_case_labels_pass(): + for label in ["Getting started", "Naming conventions", "Users & roles", + "Multi-flow workflows", "Understand Kosli", "kosli attest"]: + assert not nav_audit.is_title_case(label), label + + +def test_title_case_labels_are_flagged(): + for label in ["Naming Conventions", "Data Sources", "Helm Charts", + "Managing Environments", "Roles & Responsibilities"]: + assert nav_audit.is_title_case(label), label + + +def test_proper_nouns_and_acronyms_are_allowed_mid_label(): + for label in ["Report AWS environments", "The Kosli CLI", "Using GitHub Actions", + "Attest with Snyk", "Kubernetes and Terraform"]: + assert not nav_audit.is_title_case(label), label + + +# --- shape ------------------------------------------------------------------ + +def test_single_child_group_is_flagged(): + nav = {"tabs": [{"tab": "Docs", "groups": [{"group": "Lonely", "pages": ["a/one"]}]}]} + kinds = {f["kind"] for f in nav_audit.shape_findings(nav)} + assert "single-child group" in kinds + + +def test_deep_nesting_is_flagged_against_the_limit(): + nav = {"tabs": [{"tab": "Docs", "groups": [ + {"group": "One", "pages": [ + {"group": "Two", "pages": [ + {"group": "Three", "pages": ["a/deep"]}]}]}]}]} + deep = [f for f in nav_audit.shape_findings(nav, max_nesting=2) + if f["kind"] == "deep nesting"] + assert [f["where"] for f in deep] == ["Docs > One > Two > Three"] + assert not [f for f in nav_audit.shape_findings(nav, max_nesting=5) + if f["kind"] == "deep nesting"] + + +def test_tabs_and_menu_items_are_exempt_from_sentence_case(): + """Tabs and menu items are section names, not headings.""" + nav = {"tabs": [{"tab": "Implementation Guide", "menu": [ + {"item": "CLI Reference", "groups": [{"group": "General", "pages": ["a/one", "a/two"]}]}]}]} + assert not [f for f in nav_audit.shape_findings(nav) if f["kind"] == "Title Case label"] + + +def test_generated_subtree_is_exempt_from_shape_checks(): + """`kosli allow` having one subcommand is upstream truth, not a defect.""" + nav = {"tabs": [{"tab": "Reference", "menu": [ + {"item": "CLI Reference", "groups": [ + {"group": "kosli allow", "pages": ["client_reference/kosli_allow_artifact"]}]}]}]} + assert nav_audit.shape_findings(nav) == [] + + +def test_non_generated_subtree_is_still_checked(): + nav = {"tabs": [{"tab": "Reference", "menu": [ + {"item": "Template Reference", "groups": [ + {"group": "Templates", "pages": ["template-reference/flow_template"]}]}]}]} + kinds = {f["kind"] for f in nav_audit.shape_findings(nav)} + assert "single-child group" in kinds + + +def test_oversized_group_is_flagged(): + nav = {"tabs": [{"tab": "Docs", "groups": [ + {"group": "Big", "pages": [f"a/p{n}" for n in range(20)]}]}]} + kinds = {f["kind"] for f in nav_audit.shape_findings(nav, max_children=12)} + assert "oversized group" in kinds + + +def test_inconsistent_icons_reported_once_per_sibling_set(): + nav = {"tabs": [{"tab": "Docs", "groups": [ + {"group": "A", "icon": "book", "pages": ["a/one", "a/two"]}, + {"group": "B", "pages": ["b/one", "b/two"]}, + ]}]} + icons = [f for f in nav_audit.shape_findings(nav) if f["kind"] == "inconsistent icons"] + assert len(icons) == 1 + assert "A" in icons[0]["detail"] and "B" in icons[0]["detail"] + + +# --- CLI contract ----------------------------------------------------------- + +def _run(nav_file, root, *args): + return subprocess.run( + [sys.executable, str(_SCRIPT), "--nav-file", str(nav_file), "--root", str(root), *args], + capture_output=True, text=True, + ) + + +def test_check_exits_nonzero_on_orphan(tmp_path): + nav = {"tabs": [{"tab": "Docs", "groups": [{"group": "A", "pages": ["a/one"]}]}]} + nav_file = _write_site(tmp_path, nav, ["a/one.md", "a/lonely.md"]) + result = _run(nav_file, tmp_path, "--check") + assert result.returncode == 1 + assert "a/lonely" in result.stdout + + +def test_check_passes_on_clean_site(tmp_path): + nav = {"tabs": [{"tab": "Docs", "groups": [{"group": "A", "pages": ["a/one", "a/two"]}]}]} + nav_file = _write_site(tmp_path, nav, ["a/one.md", "a/two.md"]) + result = _run(nav_file, tmp_path, "--check") + assert result.returncode == 0, result.stderr + + +def test_check_does_not_fail_on_shape_findings_alone(tmp_path): + """Shape is advisory. A single-child group must never block a PR.""" + nav = {"tabs": [{"tab": "Docs", "groups": [{"group": "Lonely", "pages": ["a/one"]}]}]} + nav_file = _write_site(tmp_path, nav, ["a/one.md"]) + result = _run(nav_file, tmp_path, "--check") + assert result.returncode == 0, result.stderr + assert "single-child group" in result.stdout + + +def test_json_output_is_machine_readable(tmp_path): + nav = {"tabs": [{"tab": "Docs", "groups": [{"group": "Lonely", "pages": ["a/one"]}]}]} + nav_file = _write_site(tmp_path, nav, ["a/one.md", "a/lonely.md"]) + payload = json.loads(_run(nav_file, tmp_path, "--json").stdout) + assert payload["orphans"] == ["a/lonely"] + assert payload["dangling"] == [] + assert any(f["kind"] == "single-child group" for f in payload["shape"]) + + +def test_real_repo_navigation_has_integrity(tmp_path): + """The committed navigation must route every page. CLAUDE.md rule 2.""" + repo = Path(__file__).resolve().parent.parent + orphans, dangling = nav_audit.integrity_findings( + nav_audit.load_nav(repo / "config" / "navigation.json"), repo + ) + assert orphans == [], f"orphaned pages: {orphans}" + assert dangling == [], f"dangling nav entries: {dangling}" From eacf77acde2b2506d245c76475894272ae504532 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dan=20Gr=C3=B8ndahl?= Date: Thu, 3 Sep 2026 11:44:51 +0200 Subject: [PATCH 5/5] feat: add docs-restructure issue template for the doc-structure audit Restructure findings have a shape the existing backlog has no precedent for. Coverage issues ("document X") look like #364, #169, #167 and need no scaffold; a navigation change needs current structure, proposed structure, reader impact, and - the part that gets missed - whether any URLs change. Markdown, not a YAML form, deliberately. `gh issue create --body` does not apply templates mechanically, and YAML issue forms cannot be filled from the CLI at all, so a form would help humans in the web UI and do nothing for the cron job. A markdown template is something the agent can read and follow, and humans still get it in the chooser. The skill and the workflow prompt both say to read it and reproduce its sections. The template's URL impact section encodes the distinction that decides whether a restructure is safe: - renaming a group label -> no URL change - re-ordering within a group -> no URL change - collapsing a group wrapper -> no URL change if the page file stays put - moving or renaming a page -> URL changes, needs config/redirects.json Its Out of scope section names the generated subtrees, so a proposal cannot drift into reshaping `Reference > CLI Reference`. Also tells the job to cluster before filing. The current audit reports 33 shape findings that reduce to about four underlying issues - Implementation Guide alone appears under three finding kinds across both its phases. Filing one issue per line would bury the signal. Adds ISSUE_TEMPLATE/config.yml with blank_issues_enabled: true. The first template makes GitHub show a chooser and de-emphasise blank issues, and this backlog has plenty of one-line entries (#115, #134, #80) that should stay easy to file. --- .claude/skills/doc-structure/SKILL.md | 8 ++ .github/ISSUE_TEMPLATE/config.yml | 3 + .github/ISSUE_TEMPLATE/docs-restructure.md | 109 +++++++++++++++++++++ .github/workflows/doc-structure.yml | 9 ++ 4 files changed, 129 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/docs-restructure.md diff --git a/.claude/skills/doc-structure/SKILL.md b/.claude/skills/doc-structure/SKILL.md index eafff343..9b98c4b0 100644 --- a/.claude/skills/doc-structure/SKILL.md +++ b/.claude/skills/doc-structure/SKILL.md @@ -99,6 +99,14 @@ Follow the repo's conventions: - **Labels** — `content` on nearly everything. Add `documentation` for a missing or incomplete page, `enhancement` for a structural change, `automation` for a generator or workflow fix. Add `priority: high` only for something actively misleading readers. Confirm against `gh label list` rather than assuming. - **Body** — state the finding, the evidence that proves it (file paths, line numbers, the grep that found it, the changelog entries involved), and what a fix would look like. Someone should be able to act on the issue without re-running the audit. +This audit files two kinds of issue, and they need different bodies: + +**Restructure** — a navigation shape change from step 2 or 3. Read `.github/ISSUE_TEMPLATE/docs-restructure.md` and follow its sections. `gh issue create --body` does not apply a template mechanically, so you must reproduce the structure yourself. Its **URL impact** section is the one that matters: a group rename or a collapsed wrapper changes no URLs, but moving a page file does and needs a `config/redirects.json` entry. Get that distinction right or the issue will propose a change that breaks live links. + +**Coverage** — a feature from step 4 with missing or incomplete docs. No template; these match the existing backlog (an issue naming the page to change and the changelog entries that prove the gap). Say which product(s) shipped the feature, which changelog entries cover it, which page should explain it, and what a reader currently cannot find out. + +Never file a restructure issue against generated navigation. The `Reference ▸ CLI Reference` subtree mirrors the CLI's command tree — the audit script already excludes it, so a shape finding there means the exclusion needs fixing, not the navigation. + Do not assign, milestone, or set priority beyond the labels above. ## Step 7 — Summarize diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..c3754ff6 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,3 @@ +# Templates are a starting point, not a gate. Most docs issues are a sentence +# or two and should stay that way. +blank_issues_enabled: true diff --git a/.github/ISSUE_TEMPLATE/docs-restructure.md b/.github/ISSUE_TEMPLATE/docs-restructure.md new file mode 100644 index 00000000..00898d4d --- /dev/null +++ b/.github/ISSUE_TEMPLATE/docs-restructure.md @@ -0,0 +1,109 @@ +--- +name: Docs restructure +about: Propose a change to navigation shape — grouping, nesting, labels, or where a page lives +title: 'docs: ' +labels: ['content', 'enhancement'] +--- + + + +## Current structure + + + +``` +Documentation > Tutorials > Evaluation +``` + +**Pages affected:** + + + +- `tutorials/evaluate_trails_with_opa` + +## What's wrong + + + +## Why it matters + + + +## Proposed structure + +``` +Documentation > Tutorials + └─ tutorials/evaluate_trails_with_opa +``` + +## URL impact + + + +- [ ] No page files move — nav-only change, no redirects needed +- [ ] Page files move — redirects required, listed below + + + +## Out of scope + + + +## Evidence + + + +``` +$ python3 scripts/audit_navigation.py --json +``` + +## Verification + +- [ ] `python3 scripts/audit_navigation.py --check` — integrity still clean +- [ ] `python3 -m pytest tests/` — navigation integrity test passes +- [ ] `mint broken-links` — no new broken links +- [ ] Redirects added for every moved page, and the old URLs resolve diff --git a/.github/workflows/doc-structure.yml b/.github/workflows/doc-structure.yml index 7e88f461..0268f014 100644 --- a/.github/workflows/doc-structure.yml +++ b/.github/workflows/doc-structure.yml @@ -76,6 +76,15 @@ jobs: - Every issue body must carry the evidence that proves the finding — file paths, line numbers, the command that found it — so it can be acted on without re-running the audit. + - For a navigation restructure, read + `.github/ISSUE_TEMPLATE/docs-restructure.md` and reproduce its + sections in the issue body. `gh issue create --body` does not + apply templates mechanically, so you must follow it yourself. + Fill in its URL impact section correctly: a group rename or a + collapsed wrapper changes no URLs, but a moved page file needs a + `config/redirects.json` entry. + - Cluster before filing. Many shape findings share one underlying + cause and belong in one issue, not one issue per line. - If nothing clears the bar, file nothing and say so. A quiet run is a valid result.