Split docs/knowledge/ into a harvest-only tree and a notes tree, and add audit-scale group queries

[Wahbeh-Mohammad]

Problem

docs/knowledge/ mixes two kinds of content. The first kind is harvested from the source
documents. The second kind is written by hand during implementation. Nothing separates them,
and the hand-written content is not durable.

A harvested entry carries a <sub> line with a sha. The sha is a digest of the whole source
file, not of the entry. An edit to the text of an entry therefore does not change the sha.
The next harvest cannot see the edit. The harvest then regenerates the original text, or it
creates a duplicate.

The corpus also has two slots for implementation knowledge, and both are unused. The role
review holds 1 entry. The Superseded section is empty across all 39 topic files. The
channel that is actually used is an edit inside a harvested Conflicts entry.

Separately, the query surface cannot support an audit. An audit reads a complete group of
rules and checks the system against them. Today a group can only be one topic, and a
subsystem cannot be selected at all.

Evidence

Source manifest, from docs/knowledge/SOURCES.md. 48 files in three roots:

Root	Files	Entries	Role
docs/product-spec/	22	827	spec
docs/sdk-design-nodejs/	10	169	design
The styleguide repository, an absolute path on the harvest machine	16	478	styleguide

A comparison of each recorded sha256 against the current file gives 44 matches and 4
mismatches. All four mismatches sit in docs/sdk-design-nodejs/:

docs/sdk-design-nodejs/02-package-and-workspace-layout.md
docs/sdk-design-nodejs/07-pagination-sse-and-serialization.md
docs/sdk-design-nodejs/09-toolchain-and-quality-gates.md
docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md

Each of the four is a file that a phase edited to record an outcome. Phase 6c wrote the
PAGE-11 erratum into 07-…md §7.1. The tooling decisions changed 02-…md and 09-…md.
Phase 9 and Phase 10 extended the deviation register.

The specification tree has not drifted. The styleguide tree has not drifted. The design tree
is the live one.

The deviation register is the clearest case. The source is 177 lines and 18,505 bytes, and it
is a numbered ledger that later phases append to. The harvested copy is 39 lines and 6,163
bytes in 13 entries, so it holds about one third of the register. The copy is also stale:

docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md
  recorded: f9ecb6e7d87b
  actual:   1c3f96feaf95

Audit-scale query measurements:

Query	Result
--section rules over the whole corpus	937 entries, 347,813 bytes, 37 topics
--topic http-domain-model --section rules	12,046 bytes
--topic api-design --section rules	22 entries
--req HTTP	Fails. HTTP is not a canonical ID, and no prefix filter exists.

One topic is not a group. A search of the Rules section for the word "public" returns hits
in eight topics: api-design, http-domain-model, documentation, type-system,
module-organization, error-handling, tooling-and-quality-gates, and
styleguide-overview. An audit that queries api-design alone reports a clean result over an
incomplete set.

The CLI applies no cap to the output, so a group query is complete once the group is correct.

Proposed change

1. Two trees

docs/knowledge/
  harvested/     the topic files, INDEX.md, SOURCES.md   <- generated, never edited by hand
  notes/         topic files, created only when a note exists   <- written by hand

The notes tree mirrors the topic names. Both trees stay under docs/knowledge/, so a query
never leaves that directory.

Section	Tree
Rules, Constraints, Conclusions, Reference	harvested/
Superseded	notes/
Conflicts	The statement stays in harvested/. The resolution moves to notes/.

An entry in notes/ carries the role review, a source path, and a manual sha marker. The
existing pagination.md entry is the precedent. It uses sha:manual-6c-erratum.

A note overrides a harvested rule only when following the rule causes damage. The PAGE-11
entry is the one case in the corpus that meets that bar, and it moves to Superseded.

2. A register is not harvested

A description states an approach, and it changes slowly. Harvest it.

A register accumulates rows over time, and the implementation appends to it. Do not harvest
it. Point to it.

docs/sdk-design-nodejs/10-deliberate-deviations-from-the-reference-contract.md is a
register, and it becomes project documentation outside the corpus. docs/open-items.md is
also a register. The other design chapters stay harvested, because their 169 entries carry
the TypeScript mapping that the lookup skill depends on.

The corpus drops to 38 topic files and 1457 entries. One pointer entry in notes/ names the
path of the register and says to read it at the start of a phase.

3. Audit-scale queries

A group is expressed on the command line. No group definition file is added.

• --topic a,b,c --section rules already works, and it becomes the documented audit query.
• Add --prefix HTTP. It selects every entry that cites an ID of that family. The CLI already
  derives the list of prefixes from appendix C in derivePrefixes().
• Add a stable key per entry. Use the first 8 hex characters of a digest of the entry text,
  prefixed by the topic, for example api-design/3f2a91c7. The key survives a re-order and a
  re-harvest. The key changes when the rule changes, which is correct, because a changed rule
  needs a new check.

The audit loop then closes. The audit reads a group, checks the system, and writes a note for
each broken rule. The note names the rule by its key. The next audit reads the notes first and
reports which rules already carry one.

4. Checks

A structural check runs in CI and needs no external file:

1. No entry under harvested/ carries the role review.
2. No file under harvested/ holds a Superseded entry.
3. Every <sub> line under harvested/ cites one of the three source roots.

A drift check compares each sha256 in SOURCES.md against the current source file. It reports
the state. It does not gate, at least at first. The styleguide root is outside the repository,
so those 16 files are absent in CI. The check reports them as "not verifiable", never as a
failure.

Tasks

• Create docs/knowledge/harvested/ and docs/knowledge/notes/, and move the 39 topic
  files, INDEX.md, and SOURCES.md into harvested/.
• Split the 4 resolved Conflicts entries. Keep each statement. Move each resolution to
  notes/.
• Move the single review entry in pagination.md to notes/, in the Superseded
  section.
• Remove deliberate-deviations.md from the corpus. Add the pointer entry in notes/.
• scripts/knowledge.mjs: read both roots, add an origin field per entry, add
  --origin, add --prefix, and add the stable entry key. Line 19 defines the single
  root today, and lines 243 and 248 read it.
• scripts/knowledge.test.mjs: add cases for the second tree, --prefix, --origin, and
  the stability of the key.
• Add the structural check. Wire it into CI.
• Add the drift check. Run it by hand at first.
• Point the knowledge-harvest skill at docs/knowledge/harvested through its --corpus
  argument.
• Rewrite .claude/skills/knowledge-lookup/SKILL.md.
• Update CLAUDE.md. It states "39 topic files" and a byte count, and both change.

The rewrite of the skill

• The phase-start step becomes one query, --origin note, in place of three.
• Add one sentence on how to read a result. A role of spec, design, or styleguide states
  what the documents say. A role of review states what the implementation found, and it
  overrides the first.
• Add the audit query as a stated exception. The skill currently forbids a broad topic query,
  and that advice is right for a lookup and wrong for an audit.
• Record the topic set of each audit group in the skill, so an audit is repeatable.
• Remove the frozen counts. Point to --coverage and --list-topics instead.
• Shorten the section "Never read a whole topic file". It uses three paragraphs for one rule.
• Every query in the skill stays inside docs/knowledge/.

Out of scope

• A fan-out query across the phase checklists, docs/open-items.md, and the citations in
  packages/*/src. The skill queries docs/knowledge/ only.
• A gate on the drift check. The first version reports.
• A change to how the CLI reads
  docs/product-spec/appendix-c-consolidated-normative-requirement-index.md. The CLI derives
  the allowlist of the requirement IDs from that file at run time, and it cannot run without
  it.

Settled

A reference to docs/product-spec/ is normal, and the skill keeps the three references at
lines 50, 63, and 151. The limit to docs/knowledge/ applies to the query surface only. The
rewrite also completes the roll-up path with the mapping from an ID prefix to its owning
specification chapter. The prompt in the comment below carries that mapping.

[Wahbeh-Mohammad]

Execution prompt

Paste this into a new session to execute this issue.

---

Execute this issue. Read the issue body first, and treat it as the specification.

Read before you touch code

• .claude/skills/knowledge-lookup/SKILL.md — the skill to rewrite.
• scripts/knowledge.mjs — the CLI. Line 19 defines the single corpus root. Lines 243 and
  248 read it.
• scripts/knowledge.test.mjs — the existing tests for the CLI.
• docs/knowledge/SOURCES.md — the manifest of 48 sources, with a sha256 and a harvest date
  for each one.
• docs/knowledge/INDEX.md — the generated topic table. It carries a per-topic entry count
  and a conflicts count.
• CLAUDE.md — the section on the knowledge corpus. It states counts that this work changes.

Facts that are not obvious from the code

1. The sha in a <sub> line is a digest of the whole source file. It is not a digest of the
   entry. Every entry harvested from one file carries the same value. This is the reason a
   hand edit inside a harvested entry is invisible to the next harvest.
2. --topic a,b already accepts a comma-separated list. --topic api-design,assertions
   returns 26 entries across 2 topic files. No new flag is needed for a group of topics.
3. The CLI applies no cap to the output. The two .slice() calls trim the hint lists for a
   zero-match result only.
4. The CLI reads
   docs/product-spec/appendix-c-consolidated-normative-requirement-index.md at run time. It
   derives the allowlist of the requirement IDs from that file, and it throws when the file is
   absent. Do not remove that read. The rule "queries stay inside docs/knowledge/" applies
   to the query surface, not to a reference or to this internal read.
5. scripts/*.mjs is linted. See the files list in eslint.config.js.
6. bun run test:scripts runs node --test 'scripts/*.test.mjs'. Nothing in CI runs it. Run
   it by hand.
7. The knowledge-harvest skill accepts a --corpus argument. Its default is
   <cwd>/docs/knowledge/.

Order of work

Keep the tree green at every commit. Do the CLI before the move.

1. Extend the CLI first, with a fallback. Read harvested/ and notes/ when they exist.
   Fall back to the flat docs/knowledge/ when they do not. Add the origin field, the
   --origin filter, the --prefix filter, and the stable entry key. Write the failing test
   first for each one.
2. Move the files. Move the 39 topic files, INDEX.md, and SOURCES.md into
   docs/knowledge/harvested/. Create docs/knowledge/notes/.
3. Remove the fallback from the CLI, and update its tests.
4. Migrate the entries. Split the 4 resolved Conflicts entries. Move the single review
   entry from pagination.md into notes/, in the Superseded section. Remove
   deliberate-deviations.md from the corpus, and add the pointer entry in notes/.
5. Regenerate or correct INDEX.md and SOURCES.md. The corpus becomes 38 topic files
   and 1457 entries. Do not leave a count that disagrees with the tree.
6. Add the two checks. Add the structural check, and wire it into CI. Add the drift check,
   and leave it out of CI.
7. Rewrite the skill, and update CLAUDE.md.

Exact specifications

The origin values are harvested and note. Every entry carries one.

The stable entry key is <topic>/<hash>. The hash is the first 8 hexadecimal characters
of the sha256 of the entry text. Hash the text of the bullet only. Exclude the leading -
and exclude the <sub> line. Trim the trailing whitespace before hashing. Add a test that
proves the key does not change when the entry moves to a different line.

--prefix HTTP selects every entry that cites an ID of that family. Validate the prefix
against the list that derivePrefixes() builds from appendix C. Reject an unknown prefix with
the same style of message that an unknown ID produces today.

The structural check fails when any of these is true:

1. An entry under harvested/ carries the role review.
2. A file under harvested/ holds an entry in the Superseded section.
3. A <sub> line under harvested/ cites a path outside the three source roots.

The drift check compares each sha256 in SOURCES.md against the current file. It prints
three states: OK, DRIFT, and NOT VERIFIABLE. A missing file is NOT VERIFIABLE, never a
failure. The 16 styleguide sources are absent in CI, because that root is an absolute path on
the harvest machine. Four files drift today, and all four are in docs/sdk-design-nodejs/.
Confirm that the check finds exactly those four on a local run.

The rewrite of the skill

• Replace the three phase-start queries with one, --origin note.
• Add one sentence on how to read a result. A role of spec, design, or styleguide states
  what the documents say. A role of review states what the implementation found, and it
  overrides the first.
• Add the audit query as a stated exception to the rule against a broad topic query. Give the
  form --topic a,b,c --section rules and the form --prefix HTTP --section rules.
• Record the topic set of each audit group inside the skill, so an audit is repeatable. Start
  with the API group. Its rules span at least eight topics: api-design,
  http-domain-model, documentation, type-system, module-organization,
  error-handling, tooling-and-quality-gates, and styleguide-overview.
• State the audit loop. Read a group. Check the system. Write a note for each broken rule, and
  name the rule by its key. The next audit reads the notes first.
• Remove the frozen counts. Point to --coverage and --list-topics instead.
• Shorten the section "Never read a whole topic file". It spends three paragraphs on one rule.
• Keep every query inside docs/knowledge/.

The roll-up path stays, and it gains a step

A reference to docs/product-spec/ is sanctioned. Only the query surface is limited to
docs/knowledge/. Keep the three references at lines 50, 63, and 151 of the current skill.

256 of the 645 canonical IDs resolve only to an appendix-B roll-up. The corpus holds no
content for them, and the CLI already prints a warning for that case. The skill states the
grep for appendix C, and it stops there. Complete the path.

1. bun run knowledge --req <id> returns only roll-up entries, and the CLI warns.
2. grep -n '^| <ID> ' docs/product-spec/appendix-c-…md gives the canonical text. The leading
   | and the trailing space are load-bearing. A bare grep 'HTTP-1' also matches HTTP-10
   through HTTP-19.
3. Read the owning chapter. Take the Subsystem cell from the row in step 2, and match it to a
   chapter file in docs/product-spec/. The table below holds the mapping for the 19 prefixes
   that exist today. Derive a new prefix from its Subsystem cell.

Add this table to the skill. Do not make the reader derive it at each lookup.

Prefix	Chapter
SEAM	03-pluggable-seams-and-extension-model.md
HTTP	04-core-http-domain-model.md
IO	05-i-o-contracts.md
BODY	06-request-and-response-body-lifecycle.md
CTX	07-execution-context-model.md
PIPE, RECOV	08-execution-pipelines.md
RETRY	09-retry-and-resilience.md
REDIR	10-redirect-handling.md
AUTH	11-authentication.md
PAGE	12-pagination.md
SSE	13-server-sent-events-and-streaming.md
SERDE	14-serialization-serde.md
OBS	15-instrumentation-and-observability.md
CFG	16-configuration.md
TRANSPORT	17-transport-adapter-conformance-contract.md
ASYNC	18-asynchronous-runtime-adapter-contract.md
XCUT	19-cross-cutting-invariants-and-policies.md
NFR	20-non-functional-requirements-and-quality-bar.md

Verification

Run all of these, and paste the output:

bun run test:scripts
bun run lint
bun run knowledge --origin note --brief
bun run knowledge --prefix HTTP --section rules --brief
bun run knowledge --topic api-design,documentation --section rules --brief

Then run the structural check and the drift check, and paste both results.

Do not claim that the work is done until every command above has run and passed.

Rules for this task

• Write the failing test first, confirm that it fails, implement, and confirm that it passes.
• Every new file opens with // SPDX-License-Identifier: MIT on line 1.
• No changeset is needed. This work does not change a published package surface.
• Do not push a branch. Do not open a pull request. Do not run any gh command.

Out of scope

• A fan-out query across the phase checklists, docs/open-items.md, or the citations in
  packages/*/src.
• A gate on the drift check.
• Any change to how the CLI reads appendix C.