|
| 1 | +# Event-owned completion: one publication before capture integration |
| 2 | + |
| 3 | +Baseline: `90f21a5299188d54f984a5313e774c9ac48d6595`. This advances overall |
| 4 | +roadmap R5/G2, shared-authority L2/L7 and TS T1/T2. It closes an existing event |
| 5 | +writer correctness gap; it does not qualify that writer for shadow capture. |
| 6 | + |
| 7 | +## Reconcile already-delivered work |
| 8 | + |
| 9 | +Transaction-bound Markdown/lease shadow outboxes already exist (#3870). |
| 10 | +Complete source assembly moved to TS in #4967; prepared-entry source resolution |
| 11 | +and delivery moved to TS in #4968. Those owners must be reused. “Event source |
| 12 | +capture is missing” was too broad: the remaining gap is binding the event-log |
| 13 | +writer's actual commit to the existing capture lifecycle, then qualifying mixed |
| 14 | +writers and whole-Goal recovery. Source assembly is not another remaining PR. |
| 15 | +The `event_log_writer_not_bound` hold stays in bootstrap, capture and delivery. |
| 16 | + |
| 17 | +## Behavior and ownership |
| 18 | + |
| 19 | +Previously event-owned Todo completion appended successor add/claim events |
| 20 | +before it finished encoding the parent completion. A later error left runnable |
| 21 | +successors without a completed parent. Its context check also did not compare |
| 22 | +the actual event log under the append lock. Both failures reproduce on the |
| 23 | +baseline through the public completion function. |
| 24 | + |
| 25 | +- The event adapter now encodes the existing TS successor proposals, then |
| 26 | + submits successors and completion as one eager batch. Duplicate normalization |
| 27 | + and default/ownership decisions are removed from the Python successor helper. |
| 28 | +- `goals/state_event_append.ts` owns whole-batch identity conflicts, replay, |
| 29 | + sequence allocation and source-checksum admission. Python holds the existing |
| 30 | + sibling lock, supplies compact identity/hash facts and retains legacy codecs |
| 31 | + and IO. There is one planning RPC per batch, not per historical event. |
| 32 | +- Exact `list`/`tuple` batches validate fully before publication. Atomic replace |
| 33 | + plus file/directory fsync makes the event stream visible as all old or all new |
| 34 | + bytes. Prior bytes and event schemas remain unchanged. Lazy iterables and |
| 35 | + subclasses keep their per-item visibility/reentrancy contract; callers must |
| 36 | + materialize them before requesting an atomic source-bound batch. |
| 37 | +- Source drift returns the existing completion validation failure, without a |
| 38 | + successor prefix. A lost publication acknowledgement reports an uncertain |
| 39 | + outcome; read back the original Todo and retry completion. Terminal replay |
| 40 | + re-establishes log durability without generating more successors. |
| 41 | + |
| 42 | +This deliberately changes eager-batch failure/visibility semantics. The log is |
| 43 | +logically append-only, but the physical inode is replaced. Consumers must reopen |
| 44 | +it; an indefinitely open file descriptor is not a live-tail contract. No event |
| 45 | +schema version, capture gate, migration permission or provider default changes. |
| 46 | +No frontend setting changes: the public completion result and existing CLI/API |
| 47 | +route remain the entrypoints; only failure atomicity and replay are corrected. |
| 48 | + |
| 49 | +## Evidence and cost |
| 50 | + |
| 51 | +Public-entrypoint counterexamples fail on baseline and pass on this change. |
| 52 | +Validation also covers late duplicate/invalid events, source drift, concurrent |
| 53 | +process batches, pre-replace failure, post-replace fsync failure, exact retry, |
| 54 | +historical CRLF/no-final-newline preservation, both successor roles and dry-run. |
| 55 | +Existing event-only capture holds and non-Todo supervisor/read consumers remain |
| 56 | +covered. A source-projection return-value narrowing fixes an existing mypy |
| 57 | +failure without changing its runtime acceptance rules. |
| 58 | + |
| 59 | +The read-only source-copy rehearsal consumed 6,111,476 Markdown bytes and |
| 60 | +backfilled 874 events. A disposable registry's real CLI completed a synthetic |
| 61 | +Todo with three appended events in 1,531 ms; replay left bytes unchanged and |
| 62 | +source digest readback matched. It uses real record variety/volume, not live |
| 63 | +Goal configuration, execution or promotion; it does not assert complete archive |
| 64 | +capture. `examples/control_plane/event-completion-rehearsal.py` reproduces it. |
| 65 | + |
| 66 | +On the same 707,414-byte detached log, seven warm three-event batches had median |
| 67 | +11.33 ms on baseline and 28.94 ms on this change. This pays for admission and |
| 68 | +crash durability; it is not a speedup. Existing whole-log reads remain, and |
| 69 | +atomic publication adds a whole-file copy. Do not use this legacy adapter as the |
| 70 | +future high-throughput provider; retire it with its final caller after migration. |
| 71 | +No RPC budget was raised. File/SQLite/PostgreSQL stores are not changed here. |
| 72 | + |
| 73 | +## Remaining local-default delivery program |
| 74 | + |
| 75 | +The conditional estimate remains **5–8 cohesive delivery PRs**, subject to |
| 76 | +integration findings and existing open prerequisites. This transaction repair |
| 77 | +is a prerequisite within public-writer/capture closure, not grounds to subtract |
| 78 | +one complete package. Older 7–9 estimates describe earlier checkpoints. |
| 79 | + |
| 80 | +| Package | PRs | Concrete exit | |
| 81 | +| --- | --- | --- | |
| 82 | +| Public callers and executor boundaries | 1–2 | Reconcile real CLI/Turn/Chat writers and external-effect consumers; close current-proof/fence gaps and delete replaced Python rules. Reuse current leased handoff/selection/receipt work. | |
| 83 | +| Consumer and display closure (D1) | 1 | Integrate merged #4961 projection recovery, #4964 complete-source summary and #4922 snapshot paging; prove missing/stale display recovery through actual clients. Do not reimplement these owners. | |
| 84 | +| Selected SQLite profile (D2) | 1–2 | Continue #4224 and coordinate #4931: capacity, crash/restore/upgrade, lag, supported runtime/OS and applicable elapsed soak on the selected profile. Test count is not elapsed soak. | |
| 85 | +| Capture continuity and whole-Goal rehearsal (L7/L8, D3) | 1–2 | Bind actual event transactions to prepared/committed outbox identity; prove mixed writers, interruption, drain, old-writer fencing, canonical readback and fenced rollback on File/SQLite. Keep unbound holds until this passes. | |
| 86 | +| Default selection and bounded retirement (L9/T4) | 1 | New-Goal creation/settings/install select the qualified profile; migrate approved existing cohorts and delete old business writers only after final callers and compatibility windows close. Markdown import/export/rendering are not obsolete business writers. | |
| 87 | + |
| 88 | +File remains the explicit reference profile and SQLite the long-lived local |
| 89 | +candidate. PostgreSQL already has a provider and scoped service factory; real |
| 90 | +service authentication, deployment, restore/failover and capacity qualification |
| 91 | +remain a separate medium-term package. They are not prerequisites for local |
| 92 | +default selection and this PR does not qualify them. |
0 commit comments