|
| 1 | +# Authority format upgrade and File retained state |
| 2 | + |
| 3 | +The File provider retains its `AuthorityStore` contract and `file_v0` routing |
| 4 | +identity. The physical document changes from `loopx_file_authority_store_v0` to |
| 5 | +`loopx_file_authority_store_v1`. This does not promote a Goal or select a provider. |
| 6 | + |
| 7 | +## Storage and semantics |
| 8 | + |
| 9 | +| | Old File document | Current File document | |
| 10 | +| --- | --- | --- | |
| 11 | +| Each committed row | Complete projection | Checkpoint at cursors 1, 65, 129, …; exact delta otherwise | |
| 12 | +| Events, operation ID, original receipts | Retained | Retained without rewriting | |
| 13 | +| Cursor and provider revision | Logical transaction identity | Identical after physical upgrade | |
| 14 | +| Historical reads | Full stored projection | Reconstruct the same projection from nearest checkpoint | |
| 15 | +| Runtime acceptance | Migration input only | Normal reads and writes | |
| 16 | + |
| 17 | +File reuses the TypeScript state-delta codec already used by SQLite. Object-key |
| 18 | +changes and array splices preserve all JSON data, including empty and `__proto__` |
| 19 | +keys. Writers prove reconstruction. Revisions still hash the logical full |
| 20 | +transaction, previous revision and store identity. The ledger remains logically |
| 21 | +append-only even though File atomically replaces its physical envelope. |
| 22 | + |
| 23 | +Cold reads verify every retained transaction and the final head; a valid head |
| 24 | +cannot hide a corrupt old delta or receipt. Verified pagination reconstructs at |
| 25 | +most 63 predecessor deltas plus the requested page. The exact-byte cache remains |
| 26 | +bounded. File still reads/hashes and rewrites one retained file: this reduces |
| 27 | +repeated data, not asymptotic growth. Cold verification can be slower. Measure |
| 28 | +upgrade, cold verification, warm reads and steady writes separately. |
| 29 | + |
| 30 | +## Format recognition |
| 31 | + |
| 32 | +```bash |
| 33 | +loopx --format json authority-archive inspect --source /absolute/store-or-backup |
| 34 | +``` |
| 35 | + |
| 36 | +Recognition uses JSON schema tags or the SQLite file header plus database |
| 37 | +metadata and `user_version`, not filename extensions. It reports artifact kind, |
| 38 | +provider, physical format, Goal/store identity and migration route. Unknown |
| 39 | +versions and inconsistent SQLite version pairs are rejected by automatic |
| 40 | +upgrade. A recognized provider selector is only a routing record; PostgreSQL |
| 41 | +and NoKV still require their configured provider service for export. |
| 42 | + |
| 43 | +`metadata_only` recognition is not full history verification. Logical archives |
| 44 | +are verified through their complete digest/seal contract; backup packages check |
| 45 | +source bytes and lineage. Upgrade subsequently validates the complete store |
| 46 | +under its publication boundary. Multiple local stores are reported/upgraded |
| 47 | +independently, never silently chosen as a new live authority. |
| 48 | + |
| 49 | +Legacy Markdown/sidecar capture, project registry envelopes and shadow/outbox |
| 50 | +control records have different owners. They are not alternate File database |
| 51 | +encodings and are not rewritten by this command. Whole-Goal migration must use |
| 52 | +its source capture and writer-fence workflow. |
| 53 | + |
| 54 | +## Automatic upgrade and backups |
| 55 | + |
| 56 | +Normal readers and writers **do not accept the old File format**. Old parsing |
| 57 | +belongs only to migration. There is no migration-on-read or first-business-write |
| 58 | +conversion. SQLite's existing v1-to-v2 converter follows the same explicit gate. |
| 59 | + |
| 60 | +Default local installation and Windows installation run the candidate's upgrade |
| 61 | +command before launcher activation. `loopx update apply` for pip/pipx runs it |
| 62 | +after package installation and before host updates or service restart. Canary |
| 63 | +installation does not migrate stores. Source checkouts and externally managed |
| 64 | +package updates use the same explicit command: |
| 65 | + |
| 66 | +```bash |
| 67 | +# Preview selected runtime; global --runtime-root and --registry are supported. |
| 68 | +loopx --format json authority-archive upgrade |
| 69 | +# Back up and migrate known runtime roots from existing project registrations. |
| 70 | +loopx --format json authority-archive upgrade --all-known --execute |
| 71 | +# Read-only compatibility gate, also used before binary rollback. |
| 72 | +loopx --format json authority-archive upgrade --all-known --require-current |
| 73 | +``` |
| 74 | + |
| 75 | +Discovery includes File and SQLite stores in each known runtime's authority |
| 76 | +folders, including unselected shadows and File rollback documents. It does not |
| 77 | +scan arbitrary home directories. Disconnected custom runtime roots must be |
| 78 | +supplied explicitly. Selectors, registries, writer fences and execution leases |
| 79 | +are not changed by format upgrade. |
| 80 | + |
| 81 | +Each store has its own durable publication boundary: |
| 82 | + |
| 83 | +- File holds the ordinary writer lock, verifies the entire old history, encodes |
| 84 | + and decodes the target, and compares logical history digests. It saves exact |
| 85 | + source bytes plus store identity under `format-backups/<source-sha256>/`, |
| 86 | + verifies the backup and syncs directory entries before atomic replacement. |
| 87 | +- SQLite makes an online consistent backup, including committed WAL data. A |
| 88 | + disposable copy proves the backup through the actual v1-to-v2 converter. |
| 89 | + The source migration compares that history digest under `BEGIN IMMEDIATE` |
| 90 | + before adopting new tables. Concurrent source advancement aborts the upgrade; |
| 91 | + retry takes a fresh backup. It never silently discards intervening commits. |
| 92 | +- A manifest records source/target format, identity and integrity evidence. |
| 93 | + File recovery uses manifest hashes and actual source/target bytes, not a |
| 94 | + mutable completion flag. Interrupted conversion is retriable: before publish |
| 95 | + the old store remains; after publish the new store is already current. |
| 96 | + |
| 97 | +Unknown formats, corrupt history, failed backups or failed validation stop the |
| 98 | +upgrade. A multi-store upgrade can have completed earlier stores when a later |
| 99 | +one fails; the report retains those results. Retry resumes per store. It does |
| 100 | +not pretend to roll back the whole runtime or overwrite later business writes. |
| 101 | +For package-manager updates, a failed data upgrade does not undo the package |
| 102 | +installation; service activation remains blocked until repair. |
| 103 | + |
| 104 | +## Provider migration and recovery |
| 105 | + |
| 106 | +Physical format upgrade preserves provider identity and old revisions. Changing |
| 107 | +providers uses the existing portable logical archive as the interchange format: |
| 108 | + |
| 109 | +```bash |
| 110 | +loopx --format json authority-archive export --goal-id example --archive /absolute/history.ndjson |
| 111 | +loopx --format json authority-archive verify --archive /absolute/history.ndjson |
| 112 | +loopx --format json authority-archive restore --goal-id example --archive /absolute/history.ndjson \ |
| 113 | + --destination /absolute/new-isolated-store --provider sqlite --archive-sha256 DIGEST --execute |
| 114 | +``` |
| 115 | + |
| 116 | +File/SQLite exports restore into either File or SQLite. Restore creates a new |
| 117 | +provider identity/revisions while preserving logical history and receipts; it |
| 118 | +never selects the restored directory as live authority. This avoids one |
| 119 | +converter for every pair of storage formats. PostgreSQL's existing archive |
| 120 | +source contract remains unchanged; authenticated service activation, cutover |
| 121 | +and PostgreSQL destination administration are separate work. |
| 122 | + |
| 123 | +Old raw backups can be copied into an **isolated** provider directory, with their |
| 124 | +original identity and canonical filename, then upgraded and exported. Never |
| 125 | +rewrite a schema label or restore an old backup over newer acknowledged writes. |
| 126 | +Binary rollback requires the target runtime to pass `--require-current`; an old |
| 127 | +binary without that gate is not automatically activated. Data rollback and |
| 128 | +provider cutover require their own reviewed, fenced recovery operation. |
| 129 | + |
| 130 | +## Qualification and limits |
| 131 | + |
| 132 | +Tests cover legacy rejection, original historical identity/receipts, checkpoint |
| 133 | +pagination, malformed history, backup damage, interruptions around rename, |
| 134 | +competing migration processes, real SQLite backup/migration, and File/SQLite |
| 135 | +archive interchange. CLI validation uses the managed TS runtime. An authorized |
| 136 | +detached long-history snapshot additionally checks source immutability, exact |
| 137 | +backup bytes and every historical transaction/receipt against the parent. |
| 138 | + |
| 139 | +The CLI/install surfaces change; frontend and Lark business commands continue |
| 140 | +using the unchanged provider contract and need no new settings. This does not |
| 141 | +close default-provider promotion, D1–D3, PostgreSQL production qualification or |
| 142 | +Python business-owner retirement. Native Windows execution still requires its |
| 143 | +platform CI evidence; POSIX validation does not substitute for it. |
| 144 | + |
| 145 | +## 中文要点 |
| 146 | + |
| 147 | +旧 File 每次提交都复制完整状态；新格式每 64 条保留完整检查点，其余保存差量。 |
| 148 | +事件、原始回执、操作身份、版本号和历史内容不变。复用 SQLite 的 TS 编码规则， |
| 149 | +不删除 append-only ledger 抽象，也不切换 provider。 |
| 150 | + |
| 151 | +正常读写只接受新格式。安装／更新调用统一升级入口：先验证、自动备份并核对， |
| 152 | +再迁移和读回；源码开发也可显式调用 `authority-archive upgrade --execute`。 |
| 153 | +迁移解析旧数据属于升级工具，不是长期运行的旧格式兼容分支。 |
| 154 | + |
| 155 | +同 provider 的格式升级保持身份和版本；跨 provider 则用逻辑归档导出、验证、隔离 |
| 156 | +恢复，生成目标 provider 的新身份和版本，保留历史事实及原始回执。迁移数据不等于 |
| 157 | +获得执行权，恢复目录不会自动成为线上 authority。 |
| 158 | + |
| 159 | +升级失败时可能已有部分 store 完成，必须据实报告并重试，不能覆盖之后产生的写入。 |
| 160 | +备份仍是旧格式，恢复时应先在隔离目录升级。只回退二进制并不等于安全回退数据。 |
| 161 | +该方案减少重复存储和后续写入耗时，但冷校验仍验证全历史，可能更慢。 |
0 commit comments