Skip to content

perf(authority): copy journal JSON without repeated primitive allocation - #5251

Draft
LIHUA919 wants to merge 6 commits into
loopx-project:mainfrom
LIHUA919:codex/sqlite-scan-materialization
Draft

LIHUA919 wants to merge 6 commits into
loopx-project:mainfrom
LIHUA919:codex/sqlite-scan-materialization

Conversation

@LIHUA919

@LIHUA919 LIHUA919 commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Goal and current gap

Refs #4224 and RFC 7.2; follow-up to merged #4931.

A complete, source-stable matched-1m --cli report at 3009cdfbe retained 13 passed / 1 failed / 10 missing. At 100k, receipt p95 was 45.099 ms (50 ms budget), but scan-100 p95 was 251.909 ms (250 ms budget); the 10k scan observation was 337.380 ms. Those failures remain recorded.

CPU profiling identified full-projection JSON parsing and a second public-page structuredClone beside the unchanged digest work. A replay-only prototype did not consistently improve the full scan, so this patch addresses both materialization boundaries.

Delivered delta and ownership

  • Add an order-preserving strict-JSON copy entrypoint beside canonical encoding, sharing its validation walker. Mutable objects/arrays are copied; immutable primitives need no additional byte copy.
  • Use that owner for replay snapshots and public journal pages; SQLite obtains each historical projection from the verified replay snapshot.
  • Preserve keys, unknown JSON row metadata, sparse arrays, negative zero, independent mutable results, cursor/lookahead checks and all persisted proof bytes.

Owner: existing coordination codec/replay/journal boundaries. No new capability, provider, configuration option, persistent format or budget. Default File, opt-in SQLite and PostgreSQL scan callers are covered.

Acceptance evidence

  • Characterization passed before replacement; final codec/page/replay and real File/SQLite/archive/migration regression: 692 passed.
  • Real PostgreSQL 16.15, matching CI, on a fresh disposable server/database: 303 passed, 0 skipped.
  • Real source-checkout Python-to-TypeScript SQLite CLI: 5 passed; TypeScript typecheck passed.
  • Sensitivity: substituting the current head for historical projections fails the independent oracle through real SQLite scanCommitted; restored source passes.
  • Same Node 22.22.3, 1 MiB, 128-commit diagnostic fixture and 30 scan samples per arm: scan-100 p95 194.902 -> 67.940 ms. Fixture fingerprint matches. This is diagnostic evidence, not the formal 10k/100k qualification.
  • Public/private scan and whitespace checks passed. Raw profiles, reports, databases and local paths are excluded.

User entrypoints retain the same JSON fields and values; the real CLI tests exercise readback and update/failure paths. Existing frontend/Lark projections need no companion change because their store/API contract is unchanged. The future-facing pass reuses one copy/validation owner across replay and page materialization.

CI baseline follow-up

The first workflow at aabf559 exposed an existing base failure in the registry I/O census: check_contract moved from line 1014 to 1027 while its checked-in locator stayed stale. The immutable base 6643f36 and the candidate reproduce that same one-site drift on Python 3.11 and 3.12. A separate signed commit refreshes only the locator line; site identity, classification, policy, validator and I/O implementation remain unchanged. Both architecture inventories pass (9 tests), and direct census validation is clean on both Python versions. The original failed CI result is retained.

At exact PR head 64a75ca94, the complete Python Tests workflow passed, including all four Python shards and the previously failing shard 3. The PR rollup has 28 successful / 4 conditionally skipped / 0 pending / 0 failed checks. The skips are release upload, PyPI publication, Node forward compatibility and presentation; the real PostgreSQL check passed. CI run.

The complete, source-stable formal report at aabf55979 has now been independently verified: 14 passed / 0 failed / 10 missing. Scan-100 p95 is 137.646 ms at 10k and 77.918 ms at 100k (250 ms final-axis budget); receipt p95 at 100k is 31.756 ms (50 ms budget). Both 1 MiB axes, declared warm sample counts, 20 cold CLI samples per command per axis, WAL/FULL durability, source fingerprint and cleanup passed verification. The original failed report remains recorded. A separate full run remains frozen at pre-integration head 64a75ca94 with the same runtime, workload and thresholds. Its results will retain that source attribution; formal qualification of latest head 53fab7f09 is pending.

Upstream integration

Commit 748ace442 appends a signed merge of main at 738115bde, preserving all original commits. The sole conflict was the registry locator (1027 versus 1063); the resolved manifest exactly matches the upstream census. At that merge, the diff retained the original seven runtime/test files (134 additions, 7 deletions). Nine architecture tests, direct census on Python 3.11, five real SQLite CLI tests and TypeScript typecheck pass. Of 340 focused authority tests, 336 passed initially; four Python-child startup failures were reproduced as an old Node PATH, then all four passed with Node 22.22.3 explicitly selected. The original failed run is retained. Latest-head remote CI is pending. This integration adds no further copy/runtime rule; the existing shared codec owner remains sufficient, so no additional refactor is needed.

Python CI fixture repair

Commit 1df14ecc6 appends a signed test-only repair after all six failures were reproduced both on 748ace442 and unchanged main (738115bde). Open user records now have the valid nonblocking user_action type and goal response scope; global gates carry no conflicting agent continuation claim. The standard settlement finishes its ACK before the next guard owns scheduler authority. The replan test executes the emitted bounded ACK form, verifies Goal state/run accounting remain byte-identical and scheduler replay is idempotent, and still proves a newer guard rejects an old ACK. Scheduler handoff remains outside agent settlement.

Six reported cases now pass; the related Python set has 27 passed, typed scope negative cases have 10 passed, and the final ACK pair has 2 passed. Product runtime, validation rules, authority boundaries and thresholds are unchanged. Latest-head remote CI and formal qualification remain pending. The companion cleanup removes the duplicate unbound guard probe while retaining successor-selection and stale-ACK coverage.

Claim-wait concurrency proof

Latest head 53fab7f09 replaces the indirect 0.4-second recreation timing assertion with the actual lifetime-lock invariant: real Goal recreation completes while claim polling is explicitly paused, then the released claim rejects the recreated instance. The original CI result (0.652 seconds) remains recorded. Unchanged main passes the causal test. With 0.65 seconds of unrelated recreation delay, the old test fails and the new test passes; a private mutation holding the real lifetime guard during polling fails the new test. The complete attached-session instance suite has 30 passed. Only the test changes; runtime behavior and qualification budgets are unchanged. Event handshakes reuse the existing integration pattern and leave all worker threads joined. New-head CI and formal qualification remain pending.

Remaining holds

Exact-head formal 10k/100k requalification is required. Historical failures and the ten missing ledger rows remain holds; this PR does not qualify D2, resume the halted elapsed run, change defaults, migrate active Goals or grant promotion. Maintainer review/merge remains required.

中文摘要

本 PR 修复读取结果的重复投影复制,复用既有 strict-JSON 所有者并保留字段顺序、未知 JSON 元数据及可变对象隔离。真实 File/SQLite 回归 692 项、真实 PostgreSQL 16.15 回归 303 项(零跳过)、真实 CLI 5 项和类型检查通过。相同 128 笔诊断 fixture 的扫描 p95 为 194.902 -> 67.940 ms;它不替代完整正式复测。原始预算失败、十项 missing 与独立 D2/自然时间/晋升门禁均保留,交由维护者评审和合并。

最新 head 64a75ca94 的远端检查已结束:28 项通过、4 项按条件跳过、零失败;此前失败的 shard 3 与其余三组 Python 分片全部通过。冻结版本 aabf55979 的完整报告已独立验证:14 项通过、零失败、10 项 missing;10 万笔扫描 p95 为 77.918 ms、收据为 31.756 ms。合入 main 前的 64a75ca94 已启动相同参数的独立完整复测,仍不能宣称最新 head 或完整 D2 已通过。

合入 main 的 748ace442 以带 DCO 的追加合并提交解决清单冲突,原提交均保留。9 项架构测试、5 项真实 CLI、类型检查通过;340 项聚焦回归中首次通过 336 项,4 项 Node 子进程环境失败在固定 Node 22.22.3 后全部通过。最新 CI 与正式资格仍待验收。

测试修复 1df14ecc6 解决了在原 PR 与未修改 main 上均复现的六项过期测试假设;相关 Python 27 项、typed 范围负例 10 项、最终 ACK 两项均通过。用户 Todo 的合法类型/目标、ACK 的阶段边界、重放和新 guard 后旧 ACK 的拒绝均有实测覆盖。运行时规则与门槛保持原样,最新 CI 和正式资格仍待验收。

最新 53fab7f09 将 claim 等待的计时断言改为真实锁因果验证:等待暂停期间 Goal 重建必须完成,释放后旧实例 claim 必须被拒绝。30 项相关集成用例通过;额外延迟对照与真实持锁反例验证了敏感性。原 CI 失败保留,运行时和资格预算保持原样。

Signed-off-by: Lihua <1017343802@qq.com>
Signed-off-by: Lihua <1017343802@qq.com>
@mergify

mergify Bot commented Sep 28, 2026

Copy link
Copy Markdown

This pull request has merge conflicts with main and cannot be merged
until they are resolved. Please rebase or merge the base branch, @LIHUA919.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/main and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Sep 28, 2026
Signed-off-by: Lihua <1017343802@qq.com>
@mergify mergify Bot removed the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Sep 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant