Goalpost: streaming indexed recursive WARP without full-state materialization

[flyingrobots]

Plan: https://github.com/git-stunts/git-warp/blob/plan/streaming-recursive-materialization/docs/plans/streaming-indexed-recursive-warp.md
Design PR: #813

Hill

Make every git-warp operation independent of complete process-resident graph state. Roaring indexes select stable graph-local ordinals, bounded address pages resolve those ordinals to immutable CAS payloads, attachments remain streams, and graph-valued attachments recursively traverse other WARP graph bases with cycle safety, budgets, backpressure, and receipts.

Why this supersedes the immediate release roadmap

A real Think migration completed scratch rewrite and verification, promoted prepared refs, and then failed promoted-runtime verification because a 23,995,927-byte logical replay basis reached the ordinary 5 MiB CBOR decoder ceiling. Rollback succeeded and recovery refs were retained.

The failure proves that the v19 bounded-reading work left a whole-state escape hatch in retained resume, hashing, checkpoint, comparison, BTR, migration, and attachment paths. Progress UI and migration documentation cannot make that substrate safe.

Freeze v19.0.1 publication while this goalpost is open. The existing v20.0.0 milestone already owns the streaming/materialization/traversal thesis and becomes the next functional release target.

Child slices

• Eliminate production full-state materialization contracts #814 — eliminate production full-state materialization contracts
• Define retained graph manifests and bounded ordinal address pages #815 — define retained graph manifests and bounded ordinal address pages
• Remix Roaring indexes as the retained-state selection plane #816 — remix Roaring indexes as the retained-state selection plane
• Stream causal reduction and replace state hashes with root commitments #817 — stream causal reduction and replace state hashes with root commitments
• Make attachment APIs stream-only and bound all byte collection #818 — make attachment APIs stream-only and bound all byte collection
• Add typed recursive WARP graph attachment references #819 — add typed recursive WARP graph attachment references
• Traverse recursive WARP graphs with cycle-safe bounded receipts #820 — traverse recursive WARP graphs with cycle-safe bounded receipts
• Retain and diagnose recursive graph references across CAS boundaries #821 — retain and diagnose recursive graph references across CAS boundaries
• Migrate retained substrates directly into indexed graph roots #822 — migrate retained substrates directly into indexed graph roots
• Prove bounded memory across global jobs, attachments, migration, and recursion #823 — prove bounded memory across global jobs, attachments, migration, and recursion

Existing issue lineage

Corrective completion of #626–#635, #734–#742, and #758–#760.

Related open debt: #185, #186, #192, #211, #225, #226, #228, #248, and #260.

Migration UX and guidance retained from #806–#812. #810 is blocked on the retained-state architecture; #811 and #812 remain follow-on work.

Closed predecessor issues remain historical records. This issue does not silently redefine what their original PRs proved.

Release acceptance

• No production port, adapter, service, command, or proof accepts or returns a complete graph state.
• Full-state codecs, replay-basis storage, eager attachment APIs, and unbounded asset collection are deleted.
• Targeted reads and patches use Roaring selection plus bounded address/payload pages.
• Checkpoint, resume, comparison, BTR, export, migration, and diagnostics stream or use bounded external workspaces.
• State identity is a deterministic commitment over retained roots.
• Typed pinned/live graph attachments recursively traverse with cycle, sharing, budget, capability, redaction, and receipt semantics.
• Recursive retention and doctor are cycle-safe and cross-repository authority remains explicit.
• Generated oversized state and attachment fixtures pass under reviewed heap/RSS caps without checking large repositories into Git.
• A disposable copy of the real Think repository passes migration, reopen, append, readings, attachment streaming, recursive traversal, receipts, and doctor.
• Migration discovery, one-pass confirmation, progress, disk guidance, Wesley guidance, and API diagrams from Make v18-to-v19 migration discover graphs before confirmation #806–Define and satisfy the v19 migration guide requirements #810 are complete against the new representation.
• Merged-main release evidence is current and green.

Non-solution

Increasing the global CBOR limit, embedding truncated CAS hashes in Roaring integers, or renaming a complete-state object does not satisfy this goalpost.

[coderabbitai]

🔗 Related PRs

#93 - Fix recursive tree OID read fanout and release v17.0.1 [merged]
#107 - Prepare v18.0.0 release metadata and final gates [merged]
#732 - Fix: anchor state-cache payloads against Git GC [merged]
#788 - Add one-shot v18-to-v19 retained-substrate migration [merged]
#813 - Define streaming indexed recursive WARP architecture [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!