docs: advisory pivot + first live advisory run (2026-09-18)

CLAUDE.md

@@ -7,8 +7,9 @@ reverse once data and prompts accumulate.
 The schema defines every structure; pydantic models (`store/models.py`) and the
 SQLite DDL (`store/migrations/`) are both derived from it. Change the schema
 first; regenerate the models and add a migration — never the other way around.
-**`schema/extraction_schema_v4.yaml` is current**; `_v1`/`_v2`/`_v3` are retained
-unedited. v2 added per-attribute grounding (`entity.attribute_citations`) and a
+**Two domain packs share one engine: `schema/extraction_schema_v5.yaml` is the
+ADVISORY pack (current, GEI-7) and `schema/extraction_schema_v4.yaml` is the
+FOUNDRY pack (retained unedited)**; `_v1`/`_v2`/`_v3` are retained unedited. v2 added per-attribute grounding (`entity.attribute_citations`) and a
 fail-safe `unknown` `citation.location_type`; v3 added `weakly_corroborated` as a
 `corroboration.status` verdict value (a group that agrees within tolerance but has
 <2 non-suspect members — a clean number + an honestly-disclosed sparsity number)
@@ -17,7 +18,19 @@ the per-row `corr_status`/`corr_related_claim_ids` columns (a per-group verdict
 no honest per-row home) and dropped the `cmp_baseline_entity` FK. v4 (WS-2a,
 2026-08-18 gate) added the persisted `conflict` record and two-layer content
 bounds (pydantic length+charset; DDL length CHECKs — migration 0004), and
-documented the extraction-artifact fold.
+documented the extraction-artifact fold. v5 (GEI-7, 2026-09-03) is a PARALLEL
+pack, not a supersession: advisory `doc_type`s (`nvd_record | ghsa |
+vendor_advisory | cisa_kev | researcher_writeup`), `entity_type` `cve | package |
+product | advisory`, `claim_class` (`affected_range | patched_in | cvss |
+exploit_status | workaround`), a STRUCTURED `version_range` interval (the
+verbatim string lives only in `quote_span`), and `source_record_kind` — attested
+at extract from the sidecar's operator-supplied `parser_role` via
+`store.attested_kind`, NEVER taken from the model or the document body
+(migration 0005). GEI-6 locked decisions: never auto-resolve (tiers weight and
+order, no tier wins), claim-class-specific kind ranking, CVE identity is a
+grouping-key problem. Advisory docs always select `extract_advisory_v1`
+(`extract.prompts.prompt_name_for_doc_type`); `config.toml`'s `prompt_version`
+is the foundry default only.
 
 ## The claim is the atomic unit — and its integrity rules are non-negotiable
 - **Every extracted value cites a source span** — code-enforced, not trusted to
@@ -237,20 +250,27 @@ gitignored in full.
 
 ## Module map
 - `ingest/`  — fetchers + content-addressed raw storage + provenance sidecars +
-  extraction-artifact layout + `forget` quarantine (operator path AND the
-  WS-2b direct-URL network fetcher both live; HTML discovery of document URLs
-  from index pages: WS-3)
-- `extract/` — raw doc → schema records via a versioned prompt; `run_extract`
-  wires ingest (sidecar) → extractor → `persist_extraction` + artifact;
-  `run_rebuild` refolds the DB from artifacts
+  extraction-artifact layout + `forget` quarantine (operator path, the WS-2b
+  direct-URL PDF fetcher, and the GEI-14 curated advisory JSON fetcher
+  (`AdvisoryJsonFetcher`: NVD API / GHSA / CISA KEV, JSON magic instead of
+  `%PDF-`, `parser_role` into the sidecar) all live; HTML discovery of
+  document URLs from index pages: WS-3)
+- `extract/` — raw doc → schema records via a versioned prompt (selected per
+  `doc_type`: advisory types → `extract_advisory_v1`, else the configured
+  foundry prompt); `run_extract` wires ingest (sidecar) → extractor →
+  `persist_extraction` + artifact, stamping `source_record_kind` from the
+  sidecar; `run_rebuild` refolds the DB from artifacts
 - `store/`   — SQLite persistence + conflict-recording entity reconciliation;
   sole DB owner
 - `analyze/` — cross-source corroboration + divergence (tolerance grouping over
   normalized keys, suspect members excluded from the corroborated count, H1
   publisher floor, tier-blind `favored_tier` on contradictions, advisory flags,
-  conflict surfacing; derive-on-read, read-only over `store`)
+  conflict surfacing; derive-on-read, read-only over `store`). Advisory groups
+  key on `(cve_id, package_or_product, claim_class)` with SET comparison of
+  version intervals (`version_sets.py`: overlap/subset/disjoint, GEI-9) and
+  `conflict_report.py` builds the per-CVE conflict ship artifact (GEI-13)
 - `cli.py`   — thin argument layer over the above (`db init|rebuild`, `ingest`,
-  `ingest-file`, `extract`, `report`, `forget`)
+  `ingest-file`, `extract`, `report`, `conflict-report`, `forget`)
 
 ## Testing (ENFORCE posture)
 `tests/` uses a golden-fixture harness: `tests/fixtures/<doc_id>/raw.pdf` +

README.md

@@ -1,6 +1,11 @@
-# semianalyst
+# clearspec (package `semianalyst`)
 
-A personal CLI pipeline that ingests semiconductor technical releases (foundry node announcements first; conference papers and vendor whitepapers later), extracts their quantitative claims into a canonical schema where the **claim is the atomic unit** — every value cites its source span, relative claims keep their ratio and baseline instead of being converted to absolutes, and each source carries a trust tier — and (later) corroborates or flags divergence across sources.
+A truth layer, not a scanner. One engine ingests untrusted documents, extracts quote-grounded claims into a canonical schema where the **claim is the atomic unit** — every value cites its source span, relative claims keep their ratio and baseline instead of being converted to absolutes, nothing is ever overwritten, and disagreements are persisted as conflicts — then reports where sources corroborate or diverge.
+
+Two domain packs run on that engine:
+
+- **Advisory corroboration (current wedge, schema v5, `extract_advisory_v1`).** NVD vs GHSA vs vendor vs CISA KEV on a CVE × package/product: affected range, patched-in version, CVSS, exploit status. Version ranges are structured intervals compared as sets, not numeric tolerances. **The conflict is the product** — `conflict-report` is the ship artifact, never a CVE feed, and no tier ever auto-resolves a fight.
+- **Semiconductor foundry claims (original pack, schema v4, `extract_foundry_v1`).** Foundry node announcements, conference papers, vendor whitepapers; numeric claims corroborated with a ±10% ruler and source tiers.
 
 ## Quickstart
 
@@ -13,6 +18,7 @@ uv run semianalyst ingest-file paper.pdf \     # ingest a local file (operator p
     --doc-type foundry_announcement --source-tier 2 --url https://pr.tsmc.com/...
 uv run semianalyst extract                     # extract grounded claims (needs ANTHROPIC_API_KEY only when work is pending)
 uv run semianalyst report                      # cross-source corroboration + divergence + conflicts
+uv run semianalyst conflict-report [--cve-id CVE-2024-21626]   # per-CVE conflict ship artifact (advisory pack)
 uv run semianalyst forget tsmc_n2              # quarantine a poisoned doc's provenance + refold the DB
 uv run semianalyst db rebuild                  # refold the DB from retained extraction artifacts
 ```
@@ -21,4 +27,6 @@ uv run semianalyst db rebuild                  # refold the DB from retained ext
 
 GitHub Actions runs the offline suite on pull requests and `main` (`uv run pytest -m "not live"`). That job does not set `ANTHROPIC_API_KEY`, does not pass `--run-live`, and does not fetch NVD/GHSA/CISA. Refresh `tests/fixtures/*/llm_response.json` with `uv run pytest --run-live --record`; a live CI job is deferred.
 
-See `CLAUDE.md` for the architectural rules, `schema/extraction_schema_v4.yaml` for the data model (the source of truth; `_v1`–`_v3` are retained unedited), and `docs/ROADMAP.md` for current status and priority.
+Advisory sources: `config.toml` carries `[[sources]]` rows for the NVD CVE API, GHSA, and the CISA KEV feed with a `parser_role` (`cpe`/`cna`, `reviewed`/`unreviewed`, `kev`) that ingest records in the sidecar and extract turns into `source_record_kind` — the document can never self-declare its kind. Add explicit `documents = [...]` URLs to fetch; JSON is magic-checked the way PDFs are. Operator-fed advisory JSON goes through `ingest-file --doc-type nvd_record --parser-role cpe ...`.
+
+See `CLAUDE.md` for the architectural rules, `schema/extraction_schema_v5.yaml` (advisory pack) and `schema/extraction_schema_v4.yaml` (foundry pack) for the data model (the source of truth; `_v1`–`_v3` are retained unedited), `docs/gei14-parser-role-mapping.md` for the parser_role → kind table, and `docs/ROADMAP.md` for current status and priority.

docs/ROADMAP.md

@@ -7,17 +7,124 @@ history — those live where they are authoritative:
 - **Architectural rules + the Class A deferred-decision definitions:** `CLAUDE.md`.
   This file _references_ Class A items (K/G/H/…) by name; it never redefines them.
 - **Per-gate decisions and their rationale:** `docs/reviews/<date>-<workstream>-<plan|precommit>/resolution.md`.
-- **The data model:** `schema/extraction_schema_v3.yaml` (current).
+- **The data model:** `schema/extraction_schema_v5.yaml` (advisory pack, current)
+  and `schema/extraction_schema_v4.yaml` (foundry pack, retained).
 
 Keep this file honest on every workstream boundary: when a slice lands, move its
 row from *Open* to *Built*; when a Class A item is retired, strike it here and in
 CLAUDE.md in the same commit.
 
-_Last updated: 2026-08-20._
+_Last updated: 2026-09-18._
 
 ---
 
-## Current state — the pipeline is wired end to end (WS-1 + WS-2); direct-URL network fetch is live
+## Current state — the advisory pack (M0–M2) is shipped on the WS-1/WS-2 engine; first live advisory run done 2026-09-18
+
+The product wedge is now **advisory corroboration** (Linear project ClearSpec,
+PRD in Linear; `conflict-report` is the ship artifact). The foundry pack stays
+built and tested; the engine rows below apply to both packs.
+
+| Stage (advisory pack) | State | Notes |
+|---|---|---|
+| Schema v5 + migration 0005 (GEI-7) | **BUILT** | Parallel pack, v4 unedited. `claim_class`, structured `version_range`, `source_record_kind` attested from sidecar `parser_role`. GEI-6 locks: never auto-resolve; claim-class kind ranking; CVE identity is a grouping key. |
+| `extract_advisory_v1` (GEI-8) | **BUILT, frozen** | Selected per `doc_type` (`prompt_name_for_doc_type`); offline golden `tests/test_advisory_golden.py` over the GEI-10 corpus (runc, Jenkins, OpenSSH, PAN-OS, HTTP/2 rapid reset). |
+| Version-set grouping (GEI-9) | **BUILT** | `(cve_id, package_or_product, claim_class)`; overlap/subset/disjoint on intervals; cvss keeps the numeric ruler; exploit_status is enum agreement. |
+| Operator advisory ingest (GEI-11) | **BUILT** | `ingest-file` accepts JSON or PDF + `--parser-role`; kind stamped at extract. |
+| Adversarial precommit (GEI-12) | **PASSED** | `docs/reviews/2026-09-03-advisory-pack-precommit/`; `tests/test_gei12_hostile.py` (poisoned NVD-shaped JSON, forget/refold, kind stamp, sanitizer). Live advisory injection deferred — trigger: key + `--run-live` advisory case. |
+| Curated JSON ingest (GEI-14, M1) | **BUILT** | `AdvisoryJsonFetcher`: NVD API / GHSA / CISA KEV `documents = [...]` URLs, JSON magic, same WS-2b transport policy. Offline CI on `main` (PR #12). |
+| Conflict report (GEI-13, M2) | **BUILT** | `analyze.conflict_report` + `conflict-report` CLI; contradicted + related weakly_corroborated groups with bounded, sanitized quote spans per side. Dogfooded offline on replay (GEI-53). |
+| **Live advisory run (2026-09-18)** | **DONE, scratch only** | See *Live run 2026-09-18* below. Not yet a committed `@live` test — `test_gei14_json_ingest.py`'s live fetch case is unconditionally skipped. |
+
+### Live run 2026-09-18 — what the real feeds + real model showed
+
+Scratch run (not committed; `data/` under the session scratchpad), real
+feeds + real model (`claude-opus-4-8`, `extract_advisory_v1`), zero code changes.
+Watchlist: NVD API (`parser_role = "cpe"`) for CVE-2024-21626 / -23897 / -6387
+/ -3400 and the four matching GHSA records (2 reviewed, 2 unreviewed). CISA KEV
+deliberately left out (see Next, item 3).
+
+| Step | Result |
+|---|---|
+| `run_ingest` | 8/8 fetched, 0 errors, 23 s at 20 rpm. HTTPS-only + JSON magic + sidecar `parser_role` all behaved. |
+| `run_extract` | 8/8 extracted, 0 errors, 259 s (~32 s/doc). 13 entities, 45 claims, 1 `entity_conflict` (`alias_collision`). |
+| `conflict-report` | 4 CVEs, 8 contradicted groups — **but almost none are the cross-source conflicts the product is for.** |
+
+What the live output proves, in priority order (each is a fork for the next
+plan gate, none is fixed yet):
+
+1. **CVE entities are dropped 4/4 by validate** — `entity.vendor` is required
+   (`Text80`, DDL `NOT NULL`) and a CVE has no vendor "as written", so the model
+   emits `null` → `shape invalid at ('vendor',)` → every claim pointing at the
+   CVE entity (all CVSS `cve_*` claims) cascades to `references an entity absent
+   from this document's proposal`. The golden proposals were hand-written around
+   this. Fork: nullable `vendor` for `entity_type = cve` (schema v6 — v5 is
+   frozen) vs. prompt-side default.
+2. **Every `exploit_status` claim is dropped** (`shape invalid at
+   ('exploit_status',)`, 3/3 live). The prompt's shape block shows
+   `"exploit_status": null` and never lists the enum (`known_exploited |
+   no_known_exploit | disputed`), so the model invents values. Until this is
+   fixed the KEV/exploit-status surface is invisible live regardless of item 1
+   in Next. Fix is a prompt change → `extract_advisory_v2` (v1 is frozen) + the
+   golden re-recorded.
+3. **Cross-source entity identity is the real blocker.** GHSA yields
+   `pkg_runc` (package, vendor `opencontainers`, name
+   `github.com/opencontainers/runc`); NVD yields `prod_runc_cpe` (product, vendor
+   `linuxfoundation`, name `runc`). GHSA's alias `runc` collides with the NVD
+   entity's name and G3 correctly refuses the union (the one `entity_conflict`).
+   Group key `(cve_id, package_or_product, claim_class)` therefore never joins
+   the two sources: the NVD runc range sits `uncorroborated` and the GHSA range
+   only "contradicts" its own prose restatement. The golden corpus hand-aligned
+   entity ids, which hid this. Fork: a per-CVE identity layer (purl/cpe
+   normalization, or grouping on `cve_id` + normalized product token instead of
+   entity_id) — a schema/analyze decision, not a prompt tweak.
+4. **Same-document branch ranges read as contradictions.** One NVD record lists
+   per-branch intervals (PAN-OS 10.2/11.0/11.1, macOS 12/13/14, Jenkins LTS vs
+   weekly); each became its own claim and analyze reports the set as
+   `disjoint` → `contradicted`. 6 of the 8 live contradicted groups are this
+   shape. Rule needed: claims from one `(doc_id, source_record_kind)` are ONE
+   side — union their intervals before comparing — or carry a branch dimension.
+   Same for GHSA `vulnerable_version_range` vs its prose restatement (`subset`).
+5. Honest positives: grounding held (2 ungrounded product names nulled, not
+   invented); `alias_collision` fired exactly where it should; sanitizer +
+   bounds passed on real server text; idempotency/revision paths untouched.
+
+The dogfood ticket (GEI-53) proved the pipeline on replayed proposals; this run
+is the first evidence from the real model on real feeds, and it says the pack's
+next milestone is **identity + same-document semantics**, not more sources.
+
+### Next (advisory pack) — ordered
+
+1. **Plan gate for the four live forks above** (CVE vendor nullability,
+   `extract_advisory_v2` exploit_status enum, cross-source identity, same-doc
+   union). Items 1–2 are cheap and unblock measurement; 3–4 decide whether
+   `conflict-report` ever shows an NVD-vs-GHSA row on live data.
+2. **Commit a live advisory anchor.** Promote the 2026-09-18 scratch run into a
+   `@live` test (NVD + GHSA for one golden CVE → `conflict-report`) and record
+   its `llm_response.json` so the offline suite replays real advisory model
+   output, not only hand-written proposals. `test_gei14_json_ingest.py`'s live
+   fetch case is unconditionally skipped today.
+3. **CISA KEV needs a per-CVE cut before extract.** The live catalog is ~1.7 MB
+   (`known_exploited_vulnerabilities.json`) and `run_extract` sends the whole
+   raw blob to the model as one document. The GEI-14 fetcher will store it, but
+   extraction cannot honestly ground a 1.7 MB JSON in one call. Options: filter
+   the catalog to the watchlist's CVE ids at ingest (a derived blob needs its
+   own sidecar provenance — "raw bytes are stored as-is" is otherwise broken),
+   or a per-CVE KEV lookup source. Moot until Next item 1 lands (exploit_status
+   claims are dropped anyway).
+4. **NVD CNA vs CPE at one URL.** Both roles read the same NVD API response;
+   with `parser_role = "cpe"` the model still extracts the CNA range strings and
+   they get stamped `nvd_cpe` (visible live: Jenkins `patched_in` sides quote
+   CNA `status: unaffected` records under kind `nvd_cpe`). Either split the
+   record at ingest (the golden corpus split it by hand) or make the
+   prompt/validate honour the role.
+5. **Golden fixture URL typo:** `tests/test_gei11_operator_ingest.py` cites
+   `GHSA-xr7r-f8xq-rv7p` for runc; the live id is `GHSA-xr7r-f8xq-vfvv`. Offline
+   identity only (the URL is a doc_id slug and the GEI-13 golden asserts it),
+   so fix it together with the next golden re-record, not alone.
+6. Deferred, unchanged: HTML discovery (WS-3), vendor HTML pages, bridges to
+   other tools, detection-spec export.
+
+pipeline is wired end to end (WS-1 + WS-2); direct-URL network fetch is live
 
 | Stage | State | Notes |
 |---|---|---|