Move legacy plain-text renditions into canonical authority

[salmonumbrella]

What changed

Existing successful plain-text v1 records now migrate into canonical rendition builds, per-version attachments, rendition heads, and one complete lexical generation when a current or released vault opens. The cutover preserves the exact stored UTF-8 bytes and validates the replacement FTS generation against current name and text search before publishing it.

Failed, missing, invalid, or obsolete rows stay outside serving authority and remain queued for fresh extraction. The legacy cache and released source database remain recoverable but stop acting as a second live search authority.

Why

Serving from both the released plain-text cache and the new derivative catalog would make upgrades and retries vulnerable to mixed or incomplete state. Existing vaults need one deterministic cutover without rewriting their extracted text or throwing away recovery data.

Usage

No manual migration command is required. Opening an older supported vault performs the cutover automatically; the previous head keeps serving until the replacement catalog and lexical generation are complete.

Part of #176 (F6). Stacks on #185.

[roborev-ci]

roborev: Combined Review (98bd5a4)

Code review verdict: Changes requested — one High-severity data-loss issue and four Medium-severity correctness issues were found.

High

• Rendition artifact blobs can be reclaimed while still referenced
  Location: internal/store/schema.sql:651, internal/store/gc.go:127
  GC reachability considers only content_versions, despite rendition artifacts being stored as blobs. It can delete normalized-evidence and Markdown artifact files, then fail to delete the corresponding blob row because rendition_artifacts still references it. Published renditions are consequently left without physical artifact bytes.
  Fix: Treat artifact blobs and retained build-source blobs as GC roots, or collect unreachable builds and artifacts atomically before reclaiming blobs.

Medium

• Version pruning can fail due to rendition foreign keys
  Location: internal/store/schema.sql:705
rendition_attachments references content_versions without cascading cleanup, while rendition_heads prevents attachment deletion. Pruning an attached historical version or emptying trash for a migrated document can therefore fail with a foreign-key error.
  Fix: Cascade heads and attachments when their content version is deleted, or remove them explicitly within the version-prune/trash-empty transaction while preserving shared builds.

• Metadata restore does not rebuild rendition search data
  Location: internal/store/metadata.go:594
  Metadata import restores rendition segments and heads but does not rebuild the omitted lexical generation. Restored native renditions such as PDFs lack the legacy content_fts fallback and become unsearchable.
  Fix: Build and publish a lexical generation from all imported rendition segments before completing import, with backup/restore search coverage for native renditions.

• Failed or changed extraction can leave stale searchable text
  Location: internal/store/extraction.go:302
  After cutover, only successful plain-text/v1 results republish rendition authority. If a later extraction fails or uses another extractor version, the cache row changes while the prior rendition head remains, causing stale text to continue being served.
  Fix: Reconcile or revoke legacy rendition heads for every terminal extraction replacement, including failures and version changes.

• Fingerprint limits are recorded but not enforced
  Location: document/rendition_normalize.go:36
EvidenceLexicalPolicyV1.MaxUnitRunes and MaxSegmentRunes affect the fingerprint but not output construction. Each non-empty unit becomes one segment, and the caller-provided RenditionPolicy is not bound to the attachment profile, allowing oversized output under a fingerprint claiming stricter limits.
  Fix: Derive rendition construction from the attached evidence/lexical policy, enforce unit limits, split segments at MaxSegmentRunes, and validate that the producer policy matches the canonical profile.

---

Reviewers: 2 done | Synthesis: codex, 21s | Total: 11m9s

[roborev-ci]

roborev: Combined Review (0df9a2c)

High-severity storage lifecycle and compatibility gaps must be fixed before merge; three medium-severity migration, policy, and performance issues also remain.

High

• GC can reclaim live rendition blobs — internal/store/gc.go:196, internal/store/schema.sql:651
  GC treats blobs absent from content_versions as unreachable, ignoring rendition_builds.source_sha256 and rendition_artifacts.blob_hash. It retires loose files before foreign-key checks reject deletion, leaving catalog rows pointing to missing physical data. Include processing references in reachability—or transactionally remove unreachable processing authority before reclamation—and add an end-to-end GC test retaining rendition artifacts.

• Rendition foreign keys prevent version pruning and trash emptying — internal/store/schema.sql:705
rendition_attachments restricts deletion of referenced content versions, while rendition_heads restricts attachment deletion. Processed or migrated documents therefore cannot be pruned. Define coordinated cascade behavior or explicitly delete heads and attachments before content versions while safely retaining or collecting shared builds.

• Persistent layout changed without a schema-version increment — internal/store/store.go:32, internal/store/schema.sql:590
  The layout still reports schema version 3, shipped in v0.14.0. Older binaries can accept the newer database but omit processing metadata during export, allowing downgrade backup/restore to silently discard rendition authority. Increment the schema version and add the released schema-v3 fixture, reader, and deterministic JSONL cutover coverage.

Medium

• Restored non-text renditions may remain permanently unsearchable — internal/store/processing_migration.go:75
  Migration exits early when no legacy extraction rows or legacy-searchable versions exist, even when imported rendition builds and heads do. Because lexical generations are not exported, restoring an OCR-only PDF/image vault does not rebuild search data. Publish the lexical projection whenever imported processing authority exists and test backup/restore of a non-text rendition.

• Processing profile lexical limits are not enforced — document/rendition_normalize.go:45
  Rendition construction creates one lexical segment per unit and applies only NormalizePolicy; MaxUnitRunes and MaxSegmentRunes are neither consumed nor checked. Bind these limits into construction, deterministically split oversized units, and validate staged renditions against the attached profile.

• Lexical rebuilds cause quadratic work and persistent FTS growth — internal/store/extraction.go:363, internal/store/search.go:147
  Each post-cutover extraction rebuilds a complete lexical generation while retaining all prior generations and FTS rows. Batch generation publication or collect superseded generations after active readers release their leases.

---

Reviewers: 2 done | Synthesis: codex, 17s | Total: 18m25s

[roborev-ci]

roborev: Combined Review (002d8f7)

Rendition lifecycle and restore handling have one high-severity data-corruption risk and two medium-severity integrity/search issues.

High

• internal/store/schema.sql:657 — GC can corrupt active rendition artifacts. Rendition artifacts are stored in blobs, but GC treats blobs not referenced by content_versions as unreachable. It can remove loose artifact bytes before DeleteBlobRows encounters the new foreign key and fails, leaving active renditions cataloged but physically missing. Treat rendition_artifacts.blob_hash and rendition_builds.source_sha256 as GC roots during candidate selection and final deletion checks. Add an integration test that runs GC after publishing a rendition.

Medium

• internal/store/schema.sql:708 — Published rendition references can block content deletion. Attachments reference content versions without cascading or explicit lifecycle cleanup, so pruning an older processed version or emptying trash can fail when deleting the referenced content_versions row. rendition_heads can similarly prevent attachment deletion. Integrate heads and attachments into version/node deletion using correctly ordered cleanup or appropriate cascades, then reclaim unreferenced builds separately.

• internal/store/processing_migration.go:77 — Restored rendition-only vaults can lose lexical searchability. Lexical generations are not exported with processing metadata, and migration exits early when there are no legacy extracted-text rows or text-searchable versions. Restored rendition heads and segments therefore receive no lexical generation, leaving content search empty until another publication occurs. Rebuild and publish the lexical projection after metadata import, or on open whenever restored rendition authority exists, independently of legacy plain-text migration.

---

Reviewers: 2 done | Synthesis: codex, 12s | Total: 11m58s

[wesm]

rebasing

[roborev-ci]

roborev: Combined Review (2112ebc)

Verdict: High-severity manifest coverage failure and two medium-severity migration correctness issues found.

High

• internal/store/search.go:241-244 — Legacy segments are coalesced into one FTS row, but PublishRenditionAndLexicalHeads compares raw catalog segments against generation rows. Legacy text over 1 MiB can therefore cause later rendition publications to fail manifest coverage checks. Normalize segment representations consistently for both validations.

Medium

• internal/store/processing_migration.go:129-137 — Rendition heads are created only for untrashed versions. If a document is trashed during migration and later restored, its legacy FTS entry is deleted, but restore does not queue extraction or publish a rendition, causing content search to miss it. Create heads regardless of trash status or reconcile/queue renditions on restore.

• internal/store/processing_migration.go:165-180 — Migration may queue rows with successful results from newer extractor versions. Legacy v1 extraction can then overwrite newer cached results, causing a version downgrade and possible data loss. Avoid queuing newer successful versions and reject lower-version overwrites in RecordExtraction.

No concrete security vulnerabilities or security-control weakening were identified.

---

Reviewers: 2 done | Synthesis: codex, 9s | Total: 16m30s

[roborev-ci]

roborev: Combined Review (a5b105e)

Verdict: Two medium-severity migration issues require attention.

Medium

• internal/store/processing_migration.go:87-93,172-205 — Successful extraction rows from newer extractor versions are treated as protected, but no rendition is created. Because migration deletes content_fts and makes rendition search authoritative, the text becomes unsearchable while its queue entry is removed. Migrate these results into a compatible rendition authority or retain their searchable projection.

• internal/store/extraction.go:355-379; internal/store/processing_migration.go:166-169 — After a lexical head exists, a failed legacy extraction leaves the previous rendition head searchable but excludes that version from want. Later migration compares the retained head in got against an empty expectation and may fail with a serving-compatibility error, blocking unrelated migrations or startup. Define explicit last-known-good behavior by either including retained heads in compatibility validation or atomically invalidating them on failure.

---

Reviewers: 2 done | Synthesis: codex, 9s | Total: 15m31s

[roborev-ci]

roborev: Combined Review (f555bd0)

No issues found.

---

Reviewers: 2 done | Synthesis: codex | Total: 15m23s