Import Cue decisions and settled reads from contextmd.sqlite; report procedure reads when Cue says they apply #129

Activity

1. lukemaj commented on Oct 19, 2026

   

   lukemaj

   on Oct 19, 2026

   Last edited by lukemaj

   ContributorAuthor

   More actions

   What: Import Cue decisions and report comparable first-unit procedure reads, with optional-source health and isolated import failures.
   Why: Optional Cue absence must not fail host health, one bad Library file must not hide its siblings, and Cue errors must not discard unrelated sync writes.
   So what: The branch is rebased and pushed; all 911 post-rebase tests, compileall, current-head CI and independent review pass, and PR #131 is ready for review with no merge authorized.

   Closes #129

   Scope and current handoff

   Worktree /home/luke/dev/toolboxmd/agent-observer-worktrees/129-cue-import, branch feat/129-cue-import; current base fcdb91c696777d172fbf785d0059ff6844fb4e6c (merged #130), pushed head 5089a01ee3c49c60f8d517e1de0f96baf01f31cf. The user authorized this rebase after #130 merged; it is complete. The push used an explicit lease against previous remote head 262ed28330a2e2ef718808fca707b7bc35fd96de.

   The cumulative implementation imports a read-only ContextMD snapshot, keeping Cue metadata and settled successful target reads without selecting/storing prompt, window or served text. First above-threshold units are keyed by harness/session/entry/context epoch before target/outcome filtering. A first carried unit resolves to its eventual same-epoch print and starts its read window there; consumed prints are not repeats. Hidden, cut-budget and missing-target first slots remain excluded/unavailable; never-printed carries are separate. Every user message closes the window even with invalid Cue metadata, alongside known epoch changes, the 30-minute cap and watermark. Full, partial, earlier and pending evidence remain separate. Exact native links add no duplicate events; per-harness and pooled aggregates include distinct sessions, descriptive cluster intervals and explicit gaps. Optional strict fleet export retains its stored snapshot.

   Post-rebase integration:

   • The only conflict was CHANGELOG.md under Unreleased. It now has Import Cue decisions and report procedure reads (#129) #131's Added, then Changed, above fix: recover native parser evidence and successful reads (#128) #130's Fixed. Both PRs' entries are retained and the Unreleased heading occurs once.
   • Blob-equivalence checks prove all 12 upstream non-changelog files equal the merged fix: recover native parser evidence and successful reads (#128) #130 blobs and all eight Cue non-changelog files equal the previously approved 262ed28 blobs. The changelog is the only shared path; range-diff shows only the expected changelog context/order differences. No new native parser edits were authored.
   • VERSION and manifests remain unchanged. The full suite ran exactly once after rebase, then compileall passed before the lease-protected force push. Live PR head/base were verified after GitHub's push update settled.

   Previously reviewed Opus round-2 correction retained:

   • Unknown ~user target expansion raises RuntimeError; the per-file catch now records only that file's candidates as unavailable, leaving healthy siblings visible.
   • The importer handles any ordinary Exception at the Cue savepoint boundary, rolls back only the attempt, records a safe gap and lets sync commit its other writes. TypeError retains schema_drift; other exceptions, including deeply nested owner JSON's RecursionError, record unreadable_database. No exception message or source text is emitted. BaseException interruption/exit signals still roll back and propagate.
   • Two new regression tests failed first on c5b3423 with the exact RuntimeError/RecursionError and now pass. The deep-JSON sync fixture reopens the ledger to verify old Cue data and committed placement/version markers. Its 100,000 nesting depth is guarded with an independent decoder assertRaises(RecursionError); Python 3.14 accepted the initial 10,000-depth input, so that initial attempt was not counted as reproduction proof.

   Earlier Opus final-review fixes retained:

   • health still displays every Cue status. Missing database, no Cue moments and not-yet-imported evidence exit 0 when other health checks pass; only Cue schema_drift or unreadable_database add exit 4. Existing invisible-session and unavailable-scan failures remain unchanged.
   • Library file parsing catches malformed/unreadable files individually. Only candidates for the failed file become unresolved gaps; healthy sibling targets remain available.
   • The Cue importer owns a savepoint around the attempt. A sqlite3.Error rolls back only Cue changes and records unreadable_database; non-text owner JSON causing TypeError records schema_drift. The prior Cue mirror remains intact. Exception messages are not emitted because they can contain source data. sync completes and commits placement/version-refresh work, emits the Cue gap and exits 1 for these hard errors.
   • Unreleased changelog now lists default ContextMD import and the health exit policy under Changed, with no Fixed bullets for unreleased Cue development. README describes the same public behavior.

   Latest shared-file changes in cli.py are limited to health and sync exit behavior; failure isolation belongs to cue.py for both callers. Cumulative CLI/health/fleet integration remains minimal. No native adapter parsing, identity.py, VERSION or plugin manifest changes. Version/release checks remain intentionally deferred under the explicit no-version instruction. No service, live-ledger write, sync against the live ledger, merge, release or install. Website/SEO impact: none, local CLI/report only.

   Elon record

   Requirements and who asked: User: rebase #131 onto merged #130 at fcdb91c, retain both Unreleased entries in Added/Changed/Fixed order, preserve VERSION/manifests, run the full suite once and compileall, push with force-with-lease, verify new-head CI and update #129; do not merge.
   Deleted: Duplicate Unreleased headings, stale base/proof claims and any unrelated code or release changes.
   Bottleneck: Integrating the reviewed Cue change with #130's parser fixes and proving the exact new candidate.
   Checked myself: Merged #130 and origin/main SHA, actual changelog conflict, seven-commit range-diff, all 12 upstream/eight Cue file blobs, requested section order, one 911-test suite, compileall, diff checks and the explicit lease-protected push.

   Reflection at the owner: earlier tests checked successful Cue import but did not distinguish optional absence from operational failure or exercise failure after another sync component had written. Regression tests now drive the CLI exit policy, healthy sibling resolution after bad files and durable non-Cue writes after failed imports. The correction lives in code/tests/docs, without adding a duplicate shared rule or private preference.

   Exact-head proof

   python3 -m unittest discover -s tests
   ----------------------------------------------------------------------
   Ran 911 tests in 87.619s
   
   OK
   

   Only one Observer suite ran at a time, after a process guard. python3 -m compileall -q agent_observer tests and git diff --check passed. Focused feedback: 39 Cue and 19 health tests passed. Five earlier regression test methods cover all six Cue health statuses in both text/JSON, malformed JSON/UTF-8 and unreadable Library files before healthy siblings, a real SQLite abort during mirror replacement, and a real non-text owners_json source value. A separately reopened connection proves placement and version-refresh markers were committed, the prior Cue candidate mirror retained and the safe gap stored. Sync failures return 1 after completion; optional absence stays non-failing.

   Failing-first proof at fc65091: Cue tests produced five errors (three file cases lost the healthy group, plus SQLite and owner-TypeError escaping sync); health produced six failed subcases for optional statuses returning 4 instead of 0. An intermediate editing indentation error was corrected before the passing runs; it is not counted as reproduction proof. Previous exact-head reviews are not transferred to this rebased head. Current-head independent review passed with no blocking findings; the newest exact-head review/independent status is success from lukemaj. Hosted proof CI is completed/success at this exact head. The post-rebase suite ran exactly once and passed all 911 tests in 87.619s; compileall passed afterward. This supersedes 262ed28's 896-test current-base proof. Earlier source-sequence red/green tests remain as described: latest c5b3423 baseline had RuntimeError/RecursionError escaping, then 39 Cue tests passed after the fix. No fresh live ledger copy or source-data report was run for this rebase; retained live proof below remains explicitly historical.

   Retained live proof from fc65091, 2026-10-19 15:26 Europe/Warsaw

   This is the previously verified snapshot at fc650914da3109d2823b925258d4d092a2da6187, retained as historical report proof, not a new live run of this head.

   agent-observer --version printed 1.3.0 before opening the live ledger using SQLite mode=ro for a fresh backup under ~/.local/state/agent-observer-scratch/129/. Cue import and report used only that copy; no live-ledger write or sync occurred. ContextMD was read with mode=ro. No prompt, window or served text was selected, stored or printed.

   Imported 236,636 moments, 305,831 candidates and 36,574 settled successful reads. Twelve ambiguous-target candidates remain explicit gaps.

   Pooled first units:

   • holdout: 9/106 = 8.49%, 44 contributing sessions; 95% session-cluster interval 3.82% to 14.12%. Full 8, partial-only 1, earlier 21, pending 0.
   • served: 349/809 = 43.14%, 429 contributing sessions; 95% session-cluster interval 39.30% to 47.19%. Full 225, partial-only 124, earlier 102, pending 7.

   Observed assignment sessions: holdout 44, served 432.

   Claude first units:

   • holdout: 1/9 = 11.11%, 7 contributing sessions; 95% session-cluster interval 0.00% to 37.50%. Full 1, partial-only 0, earlier 1, pending 0.
   • served: 73/225 = 32.44%, 107 contributing sessions; 95% session-cluster interval 25.71% to 40.27%. Full 39, partial-only 34, earlier 25, pending 3.

   Observed assignment sessions: holdout 7, served 107.

   Codex first units:

   • holdout: 6/86 = 6.98%, 27 contributing sessions; 95% session-cluster interval 2.13% to 12.35%. Full 6, partial-only 0, earlier 20, pending 0.
   • served: 196/453 = 43.27%, 227 contributing sessions; 95% session-cluster interval 38.37% to 47.70%. Full 126, partial-only 70, earlier 69, pending 2.

   Observed assignment sessions: holdout 27, served 227.

   Grok first units:

   • holdout: 2/11 = 18.18%, 10 contributing sessions; 95% session-cluster interval 0.00% to 45.45%. Full 1, partial-only 1, earlier 0, pending 0.
   • served: 79/128 = 61.72%, 92 contributing sessions; 95% session-cluster interval 52.38% to 70.08%. Full 59, partial-only 20, earlier 8, pending 2.

   Observed assignment sessions: holdout 10, served 95.

   Opencode first units:

   • served: 1/3 = 33.33%, 3 contributing sessions; 95% session-cluster interval 0.00% to 100.00%. Full 1, partial-only 0, earlier 0, pending 0.

   Observed assignment sessions: served 3.

   The current snapshot has grown since the earlier review. Grok has 126 resolved carry prints in the served headline and seven never-printed/expired first carry units excluded across six sessions; two direct first injections make its served denominator 128. No hidden first units occur in this snapshot, but exclusion remains fixture-tested. Other excluded first units: four not_served, outside the headline. Repeats: served 342/7,433 and holdout 8/762, outside the headline. Missing-metadata boundaries close 75 served and 10 holdout headline windows.

   Intervals are descriptive 95% percentile bootstraps of whole observed harness sessions, 2,000 resamples with a fixed seed. Small or all-equal samples can give unstable/degenerate intervals; fewer than two sessions gives none. The 44 holdout sessions, and especially individual harness holdout counts, limit precision. Reads do not prove following a procedure or model receipt, and session-wide holdout does not establish a per-procedure causal effect.

   An independent SQL row_number oracle selected first units before outcome/target filtering, found eventual same-epoch carry prints and recomputed headline windows from their delivery times. It matched pooled and every harness numerator, denominator, distinct contributing session count, resolved-carry count and unprinted count. A stable sample of 75 units, including 25 additional carry resolutions, matched 150 candidate/moment comparisons and 36 successful reads against the live read-only source tables. The oracle used only in-memory temporary metadata indexes; individual rows and paths are not published. Per-harness sum equals the pooled headline, including OpenCode (no holdout sample).

   Only the pooled and Codex observed differences are resolved in this evidence. Claude (1/9) and Grok (2/11) holdouts are too small to support per-harness claims; their rows describe the sample and harness mix only. No per-harness claim is made for OpenCode, which has no holdout sample. This is descriptive read evidence, not a causal effect.

   Delivery and retention

   Rebased onto merged #130, full local post-rebase proof passed and the branch was pushed with force-with-lease. PR #131 remains open and unmerged; exact-head CI and independent review pass, and it is ready for review. No VERSION or manifest changes, release or live-ledger operation. The prior scratch ledger and sidecars remain deleted. Worktree/branch remain for review and integration; the merging planner owns retirement.
2. lukemaj

   added 2 commits that reference this issue on Oct 19, 2026

   feat(cue): import decisions and report procedure reads (#129)

   066ea5a

   Import Cue decisions and report procedure reads (#129) (#131)

   ...

   04929e6
3. lukemaj

   closed this as completedin #131on Oct 19, 2026
4. lukemaj

   added a commit that references this issue on Oct 19, 2026

   chore(release): prepare 1.4.0 (#137)

   ...

   eda2c8d