Back up and restore document derivatives without providers

[salmonumbrella]

What changed

Backups now include the complete catalog-authorized document authority: current source versions, rendition artifacts, and staged rendition-build sources. Snapshot metadata says which derivative classes are included, locally reconstructible, or provider-dependent, with deterministic counts, bytes, blob totals, and checksums.

Restore validates the logical and physical authority after materializing loose or packed bytes, then rebuilds the lexical projection locally before any restored head can publish. Missing, corrupt, or inconsistent data leaves an existing target unchanged.

Why

The catalog was durable inside one vault, but backup still treated its bytes as generic storage. That could retain sensitive provider output without disclosing it or restore a vault whose catalog, blobs, and search head did not agree.

Usage

Use the existing backup and restore commands. Derivative authority is included and reported automatically, and restore never calls a rendition or embedding provider.

Part of #176 (F7). Stacks on #186.

[roborev-ci]

roborev: Combined Review (4be8dea)

High-severity backward-compatibility break and three medium-severity correctness issues found.

High

• internal/backupapp/app.go:197 — Legacy v1 backups may not restore faithfully. The unchanged docbank-metadata-jsonl-v1 restore adapter now filters blob paths and statistics through BackupBlobAuthorityCTE. Older v1 snapshots captured every blobs row, so snapshots containing an unreferenced or pruned blob will no longer match their recorded attachment counts. Version the scoped-backup contract or add an explicit scope marker, retain legacy behavior for pre-change v1 snapshots, and add compatibility coverage for an old snapshot containing an unreachable blob.

Medium

• internal/backupapp/metadata.go:140 — Valid provider artifacts can cause backup creation to fail. The derivative class allowlist uses image, markdown, structured, and transcript, but valid catalog roles are provider_image, provider_markdown, structured_evidence, and provider_transcript. Map the role constants to the intended class names or use their exact persisted values, with behavioral coverage for every provider artifact role.

• internal/store/schema.sql:708 — Renditions can block pruning and trash deletion. Rendition attachments reference content versions without delete handling, while rendition heads block attachment deletion. Cascade deletion from content versions to attachments and from attachments to heads, or explicitly remove both dependent records in every content-version deletion path.

• internal/store/search.go:129 — A stale lexical generation can remove newer builds from search. Reuse validates only the generation’s stored manifest, not whether it represents the complete current rendition catalog. Publication can therefore replace the global lexical head with an older generation. Compare reused and published generations against the full current catalog manifest within the publication transaction, rejecting stale generations after catalog growth.

---

Reviewers: 2 done | Synthesis: codex, 11s | Total: 12m6s

[roborev-ci]

roborev: Combined Review (ad444f6)

High-severity GC corruption risk and three medium-severity policy, normalization, and concurrency issues remain.

High

• GC omits authoritative rendition and build-source blob references — internal/store/search.go:98, internal/store/gc.go:196
  GC considers only content_versions, so it can classify derivative-only blobs—and retained build sources after version pruning—as unreachable. Loose files are removed, but blob-row deletion then fails due to the new foreign keys, leaving retained builds physically corrupted.
  Fix: Add rendition_artifacts.blob_hash and rendition_builds.source_sha256 to paged and unpaged reachability queries. Add coverage for GC after artifact publication and version pruning.

Medium

• Publication does not enforce artifact retention/request policy — internal/processing/artifacts.go:181
  Builds can publish and back up provider Markdown, typed provider artifacts, or sanitized Markdown even when the attachment profile disables retention or never requested the artifact.
  Fix: Decode the canonical attachment profile before writing payloads and reject roles inconsistent with RequestedArtifacts and RetentionDisclosure. Enforce the same invariant during metadata validation/import.

• Rendition construction can violate declared lexical limits — document/rendition_normalize.go:36
BuildRenditionV1 creates one segment per non-empty unit without applying EvidenceLexicalPolicyV1, allowing units and segments to exceed MaxUnitRunes and MaxSegmentRunes despite a fingerprint claiming those limits.
  Fix: Bind the policy into construction or validate before publication, enforcing unit bounds and deterministically splitting segments at MaxSegmentRunes.

• Lease acquisition can race with generation pruning — internal/store/search.go:682
  A concurrent AcquireLexicalGeneration can register the old committed head after publication snapshots its keep-set but before commit, while publication deletes that generation. The returned lease then pins rows that no longer exist.
  Fix: Serialize generation acquisition with publication pruning through commit and include all pre-existing leases in the keep-set.

---

Reviewers: 2 done | Synthesis: codex, 14s | Total: 15m54s

[roborev-ci]

roborev: Combined Review (49567bb)

Verdict: Changes requested — 1 high-severity and 2 medium-severity issues could compromise blob integrity and rendition correctness.

High

• GC can delete authoritative rendition blobs — internal/store/schema.sql:617
  GC treats blobs absent from content_versions as unreachable, despite rendition_builds.source_sha256 and rendition_artifacts.blob_hash now being authoritative foreign-key references. It can remove loose bytes before DeleteBlobRows encounters the foreign-key constraint, leaving metadata pointing to missing physical content and causing GC to fail.
  Fix: Exclude both rendition reference columns from GC candidates using the same authority closure as backups, or transactionally prune unreferenced builds and artifacts before retiring their blobs.

Medium

• Published rendition profiles do not enforce their fingerprinted lexical limits — document/rendition_normalize.go:41
  Rendition construction uses only fixed NormalizePolicy limits and emits one lexical segment per unit. It does not apply EvidenceLexicalPolicyV1.MaxUnitRunes or MaxSegmentRunes, so output may violate the limits claimed by its profile.
  Fix: Bind the evidence/lexical policy to RenditionPolicy, enforce unit limits, split text into bounded half-open segments, and validate the limits when attaching the canonical profile.

• Extraction can leave stale rendition authority active — internal/store/extraction.go:303
  After a lexical head exists, search ignores content_fts, but only an exact plain-text/v1 success republishes rendition authority. A successful future extractor version—or a failed v1 re-extraction—can update the ignored cache and clear its queue entry while stale text remains served through the old rendition head.
  Fix: Route every post-cutover terminal extraction through rendition publication, publish successful current-version output, withdraw superseded heads on failure, and clear queued work only after the authority transition succeeds.

---

Reviewers: 2 done | Synthesis: codex, 12s | Total: 14m40s

[roborev-ci]

roborev: Combined Review (d1518b2)

High-severity schema compatibility and garbage-collection correctness issues must be resolved before merging.

High

• internal/store/store.go:32 — Storage schema version was not bumped. Durable processing tables were added while currentStorageSchemaVersion remains at 3. Released v0.12/v0.13 binaries may accept the modified database as compatible, ignore the new authority, and mutate, back up, or garbage-collect it incorrectly. Bump the schema version and add an exact released-schema-v3 fixture plus a JSONL cutover adapter so older binaries reject the new layout.

• internal/store/schema.sql:614 — GC does not account for rendition blob references. Rendition builds and artifacts now pin blobs independently of content_versions, but GC and version-prune reachability still check only content_versions. This can delete loose bytes for derivative artifacts or unattached build sources before row deletion fails on foreign keys, leaving retained rendition records with missing physical content. Include rendition_builds.source_sha256 and rendition_artifacts.blob_hash in every liveness predicate—preferably through one shared authority query—and revalidate reachability before retiring physical data.

---

Reviewers: 2 done | Synthesis: codex, 12s | Total: 12m36s