Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -166,3 +166,41 @@ decision completeness and existing drill-down contracts at their shared typed
owner; do not infer that backend switching alone fixes these costs. A/C still
need integrated execution/adoption evidence, and no D2 elapsed soak starts or
legacy-writer deletion is certified by this follow-up.

### Read-cost qualification update

After #4931 and #5215 integrated, matched detached File/SQLite copies retained
379 original commits and the same final projection hash. On Node 24.21.0,
three fresh processes per provider measured File head reads at 5.98–6.32 s
versus SQLite at 34.5–36.0 ms; repeated reads were 9.1–10.2 ms and 25.7–28.2 ms
respectively. This is process-cold, not OS-cache-cold: File proves its entire
retained journal, whereas SQLite reads current state without making the same
full-history proof. It is evidence for a long-history SQLite candidate, not
equivalent integrity-work throughput or release-default acceptance.

Alternating two unchanged File stores exposed singleton proof-cache eviction:
every read cost 6.30–6.49 s. A bounded four-store working set keeps the first
proof for each store (6.15–6.16 s) and subsequent alternation at 9.8–11.2 ms,
with identical cursors/hashes. Exact-byte and identity checks remain mandatory;
eviction and corruption regressions cover the changed cache boundary.

Quota observation reused the existing should-run compactors: a captured single
Goal row serialized from 1,252,747 to 78,688 UTF-8 bytes, with explicit full
detail restoring the original row. This is a display measurement; collection,
decision inputs and first-read verification are not reduced by it.

A separate 148-second isolated run appended 12 commits per provider through
fresh processes, crossing a checkpoint and checking original-receipt replay,
changed-intent rejection and projection/hash parity at every step. It qualifies
that bounded storage journey, **not** Host execution, live Goal adoption or D2's
ten-day soak. No active authority, release default or legacy-writer deletion
decision changes. B still needs sustained workload/platform/capacity evidence;
C still needs consumer/onboarding and supported upgrade acceptance.

The next B slice is the remaining whole-command cold path: profile history
artifact lookup, active-contract validation and public-boundary scanning on
the same retained inputs before selecting the owning repair. Separate provider
head-read time from caller work; preserve freshness, full decision inputs and
corruption rejection. Re-run installed CLI consumers after integration. Do not
count this read optimization as closing A/C or use a fixed remaining-PR estimate;
retire a writer only with its last supported caller and recovery acceptance.
Original file line number Diff line number Diff line change
Expand Up @@ -129,3 +129,33 @@ settlement 链路验收。
history,status/quota 仍可能生成数 MB 诊断包。在现有共享 typed owner 保留决策
完整性与 drill-down 合同,不能推断换后端就能消除这些成本。A/C 仍需执行/采用集成
证据;本轮没有启动 D2 自然时间 soak,也没有认证旧 writer 可以删除。

### 读取成本验收更新

#4931、#5215 集成后,配对的 File/SQLite 隔离副本保留 379 笔原始提交,最终
projection hash 相同。Node 24.21.0 下,每个 provider 分别启动三个新进程,
File 首次 head 读取为 5.98–6.32 秒,SQLite 为 34.5–36.0 毫秒;后续读取分别为
9.1–10.2 毫秒、25.7–28.2 毫秒。这是进程冷读,没有清空 OS 文件缓存;File
验证全部保留历史,SQLite 读取当前状态,不承担相同的全历史证明。这支持将 SQLite
作为长历史候选,但不是同等完整性工作量的吞吐比较,也不构成发布默认值验收。

交替读取两个未变化的 File 存储,暴露了单份证明缓存互相淘汰的问题:每次都要
6.30–6.49 秒。改为有总容量上限的四份缓存后,各存储首次验证仍为 6.15–6.16 秒,
后续交替读取为 9.8–11.2 毫秒,cursor/hash 相同。每次仍检查实际字节摘要和存储
身份;淘汰与损坏回归覆盖缓存边界。

Quota 观察复用既有 should-run 摘要:捕获的单 Goal 行序列化由 1,252,747 降至
78,688 UTF-8 字节,显式明细恢复原行。这是展示体积测量,未减少采集、决策输入或
首次读取的验证成本。

另一项 148 秒隔离演练通过新进程为两种 provider 各追加 12 笔提交,跨越 checkpoint,
逐轮验证原回执重放、变更意图拒绝及 projection/hash 一致性。它证明这段有界存储
流程,**不代表** Host 执行、活跃 Goal 采用或 D2 的十天 soak 已完成。本次不改变活跃
authority、发布默认值或旧 writer 删除决定。B 仍缺持续负载/平台/容量证据;C 仍需
consumer/新建入口及受支持升级验收。

B 的下一段是剩余整命令冷路径:在相同保留输入上分别分析历史 artifact 查找、
active-contract 验证和公共边界扫描,再选择所属 owner 修复。区分 provider head
读取与调用方工作,保留 freshness、完整决策输入和损坏拒绝;集成后重跑安装态 CLI
消费者。不能把读取优化计为 A/C 完成,也不继续给固定的剩余 PR 数;只有最后受支持
调用方退出且恢复验收通过,才能删除对应 writer。
18 changes: 18 additions & 0 deletions docs/quota-allocation.md
Original file line number Diff line number Diff line change
Expand Up @@ -890,6 +890,24 @@ The first screen should make it obvious why a project is quiet:

## CLI Surface

`quota status` and `quota plan` now default to bounded Todo summaries in JSON,
reusing the summaries already used by `quota should-run`. Previously these two
observation commands returned full Todo lists. Counts, quota decisions, ordering
and health remain intact; `payload_compaction` identifies omitted lists and their
detail command. Planning still consumes complete input before this CLI projection.
Consumers that read individual Todo metadata or every item must opt into detail:

```bash
loopx --format json quota status --include-detail all
loopx --format json quota plan --include-detail agent-todos --include-detail user-todos
```

Keep the original registry, runtime and Goal selection when following a detail
command. `all` expands only the sections supported by that command. Detail reads
do not acquire a Turn or spend quota. Markdown plan rendering and standalone
`status`/`todo list` are unchanged. This bounds Todo-list display growth, not the
cost of gathering and verifying the input or the total number of Goals returned.

The first read-only or preview commands are:

```bash
Expand Down
12 changes: 8 additions & 4 deletions docs/reference/contracts/interface-budget-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,15 @@ removed without a separately validated caller migration.
| `evidence-log --thin --limit 5` | explicit-limit cold path | returned-evidence bound | referenced run-history and rollout-event artifacts |

`quota should-run` uses one repeatable cold-path selector:
`--include-detail scheduler`, `agent-todos`, `user-todos`, or
`goal-boundary`; `--include-detail all` expands every section. Public docs,
`--include-detail scheduler`, `agent-todos`, `user-todos`, `vision`, or
`goal-boundary`. `quota status` and `quota plan` accept `agent-todos` and
`user-todos`; `quota monitor-poll` accepts `decisions`.
`--include-detail all` expands the selected command's sections. Public docs,
emitted `detail_ref` commands, and internal callers use only this selector.
Unknown sections and selectors attached to another quota command fail before
status collection.
Unknown or unsupported sections fail before status collection, including when
combined with `all`. Status/plan summaries preserve counts and decisions and
declare omitted lists; explicit detail preserves the full Todo metadata. These
are CLI display projections after full planning, not truncated provider inputs.

The canonical emitted-output inventory and current characterization ceilings
live in `loopx.control_plane.testing.cli_output_budget`. Those ceilings are
Expand Down
9 changes: 7 additions & 2 deletions docs/reference/file-authority-state-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,13 @@ append-only even though File atomically replaces its physical envelope.

Cold reads verify every retained transaction and the final head; a valid head
cannot hide a corrupt old delta or receipt. Verified pagination reconstructs at
most 63 predecessor deltas plus the requested page. The exact-byte cache remains
bounded. File still reads/hashes and rewrites one retained file: this reduces
most 63 predecessor deltas plus the requested page. The exact-byte cache retains
at most four store paths in least-recently-used order, with a shared 128 MiB
serialized history/read-view budget. A large journal can retain only its head
and receipt index (up to 16 MiB per read view); scans and writes still verify its
history. Every cache hit requires matching file digest and store identity, not
only file timestamps. These are encoded-data bounds, not a heap/RSS limit.
File still reads/hashes and rewrites one retained file: this reduces
repeated data, not asymptotic growth. Cold verification can be slower. Measure
upgrade, cold verification, warm reads and steady writes separately.

Expand Down
3 changes: 3 additions & 0 deletions loopx/cli_commands/quota.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
from ..control_plane.capability_hooks import InteractionProjectionHookRegistration
from ..control_plane.quota.cli_projection import (
compact_quota_monitor_poll_cli_payload,
compact_quota_plan_cli_payload,
compact_quota_should_run_cli_payload,
)
from ..control_plane.quota.effective_action import EffectiveAction
Expand Down Expand Up @@ -318,6 +319,8 @@ def _project_quota_cli_payload(
instead of masking it with a crash (issue #3687).
"""
if not bool(getattr(args, "turn_envelope", False)):
if args.quota_command in {"status", "plan"}:
return compact_quota_plan_cli_payload(payload, detail_sections=detail_sections)
if args.quota_command == "should-run":
return compact_quota_should_run_cli_payload(
payload,
Expand Down
17 changes: 6 additions & 11 deletions loopx/cli_commands/quota_context.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,7 @@
from ..status import AUTONOMOUS_REPLAN_PERIODIC_LOOKBACK, collect_status
from ..turn_identity import mint_turn_instance_id, normalize_turn_instance_id
from .quota_request import (
QUOTA_MONITOR_POLL_DETAIL_SECTIONS,
QUOTA_SHOULD_RUN_DETAIL_SECTIONS,
QUOTA_COMMAND_DETAIL_SECTIONS,
quota_detail_sections_from_args,
validate_quota_command_request,
)
Expand Down Expand Up @@ -124,17 +123,13 @@ def validate_quota_command_context_request(
"--turn-envelope is only valid with `quota should-run`"
)
requested_details = set(getattr(args, "include_details", None) or ())
if requested_details and command not in {"should-run", "monitor-poll"}:
if requested_details and command not in QUOTA_COMMAND_DETAIL_SECTIONS:
raise QuotaCommandValidationError(
"--include-detail is only valid with `quota should-run` or "
"`quota monitor-poll`"
)
if requested_details and "all" not in requested_details:
allowed_details = set(
QUOTA_MONITOR_POLL_DETAIL_SECTIONS
if command == "monitor-poll"
else QUOTA_SHOULD_RUN_DETAIL_SECTIONS
"--include-detail is only valid with `quota status`, `quota plan`, "
"`quota should-run` or `quota monitor-poll`"
)
if requested_details:
allowed_details = {*QUOTA_COMMAND_DETAIL_SECTIONS[command], "all"}
unsupported_details = sorted(requested_details - allowed_details)
if unsupported_details:
raise QuotaCommandValidationError(
Expand Down
3 changes: 2 additions & 1 deletion loopx/cli_commands/quota_registration.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ def register_quota_command(
action="append",
choices=[*QUOTA_DETAIL_SECTIONS, "all"],
help=(
"Include one command-specific cold-path detail section. For `quota "
"Include one command-specific cold-path detail section. Status/plan default "
"to bounded Todo summaries; use agent-todos or user-todos for full lists. For `quota "
"should-run`: scheduler, agent-todos, user-todos, goal-boundary, or "
"vision. For `quota monitor-poll`: decisions. Repeat for multiple "
"sections or use `all`."
Expand Down
11 changes: 8 additions & 3 deletions loopx/cli_commands/quota_request.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,13 @@
"vision",
)
QUOTA_MONITOR_POLL_DETAIL_SECTIONS = ("decisions",)
QUOTA_PLAN_DETAIL_SECTIONS = ("agent-todos", "user-todos")
QUOTA_COMMAND_DETAIL_SECTIONS = {
"status": QUOTA_PLAN_DETAIL_SECTIONS,
"plan": QUOTA_PLAN_DETAIL_SECTIONS,
"should-run": QUOTA_SHOULD_RUN_DETAIL_SECTIONS,
"monitor-poll": QUOTA_MONITOR_POLL_DETAIL_SECTIONS,
}
QUOTA_DETAIL_SECTIONS = (
*QUOTA_SHOULD_RUN_DETAIL_SECTIONS,
*QUOTA_MONITOR_POLL_DETAIL_SECTIONS,
Expand Down Expand Up @@ -178,9 +185,7 @@ def quota_detail_sections_from_args(args: argparse.Namespace) -> frozenset[str]:
sections.add("scheduler")
if "all" in sections:
sections.update(
QUOTA_MONITOR_POLL_DETAIL_SECTIONS
if args.quota_command == "monitor-poll"
else QUOTA_SHOULD_RUN_DETAIL_SECTIONS
QUOTA_COMMAND_DETAIL_SECTIONS.get(args.quota_command, ())
)
sections.discard("all")
return frozenset(sections)
40 changes: 28 additions & 12 deletions loopx/control_plane/coordination/file_authority_store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,15 @@ import {AuthorityJournalScan} from "./authority_journal_scan.ts";

const STORE_IDENTITY_PATTERN = /^file:[0-9a-f]{32}$/;
// File retains a checkpoint/delta journal in one durable envelope. A managed Effect server
// opens a new store handle for each request. Keep one verified read view across
// handles, keyed by exact bytes and store identity. Large journals retain only
// opens a new store handle for each request. Retain a bounded working set across
// handles so alternating Goals do not evict each other on every observation.
// Every lookup still reads and hashes the full file and checks store identity. Large journals retain only
// the head and receipt index in memory; commits and scans still load and verify
// the complete history. This is a bounded read optimization, not a new source
// of authority or a substitute for the SQLite long-goal profile.
const MAX_CACHED_DOCUMENT_BYTES = 128 * 1024 * 1024;
const MAX_CACHED_READ_VIEW_BYTES = 16 * 1024 * 1024;
const MAX_CACHED_STORES = 4;
interface VerifiedDocument {
path: string;
identity: string;
Expand All @@ -49,7 +51,7 @@ interface VerifiedDocument {
}>;
document?: FileAuthorityJournal;
}
let verifiedDocument: VerifiedDocument | null = null;
const verifiedDocuments = new Map<string, {view: VerifiedDocument; bytes: number}>();
// Only identical immutable input bytes share in-flight verification. Failed
// proofs are removed too; neither a path nor a pending promise grants authority.
const pendingVerification = new Map<string, Promise<FileAuthorityJournal>>();
Expand All @@ -76,10 +78,22 @@ function rememberVerifiedDocument(path: string, identity: string, raw: Uint8Arra
const view: VerifiedDocument = {path, identity, digest, head: document.head,
providerRevision: document.provider_revision, cursor: document.cursor,
receipts, document};
verifiedDocument = raw.byteLength <= maxDocumentBytes ? view
: viewBytes <= MAX_CACHED_READ_VIEW_BYTES
? {...view, document: undefined}
: null;
// Account for serialized history and the separate head/receipt index. This
// is a retained-byte bound, not a claim about the JS heap or process RSS.
const fullBytes = raw.byteLength + viewBytes;
const retainHistory = raw.byteLength <= maxDocumentBytes && fullBytes <= MAX_CACHED_DOCUMENT_BYTES;
const retained = retainHistory ? view : {...view, document: undefined};
const bytes = retainHistory ? fullBytes : viewBytes;
verifiedDocuments.delete(path);
if (retainHistory || viewBytes <= MAX_CACHED_READ_VIEW_BYTES) {
verifiedDocuments.set(path, {view: retained, bytes});
let total = [...verifiedDocuments.values()].reduce((sum, entry) => sum + entry.bytes, 0);
while (verifiedDocuments.size > MAX_CACHED_STORES || total > MAX_CACHED_DOCUMENT_BYTES) {
const oldest = verifiedDocuments.keys().next().value!;
total -= verifiedDocuments.get(oldest)!.bytes;
verifiedDocuments.delete(oldest);
}
}
return view;
}

Expand Down Expand Up @@ -274,10 +288,12 @@ export class FileAuthorityStore implements AuthorityStore {
const identity = knownIdentity ?? await this.readStoreIdentity();
try {
const digest = documentDigest(raw);
if (verifiedDocument?.path === this.path &&
verifiedDocument.identity === identity && verifiedDocument.digest === digest &&
(!requireHistory || verifiedDocument.document !== undefined)) {
return verifiedDocument;
const cached = verifiedDocuments.get(this.path);
if (cached?.view.identity === identity && cached.view.digest === digest &&
(!requireHistory || cached.view.document !== undefined)) {
verifiedDocuments.delete(this.path);
verifiedDocuments.set(this.path, cached);
return cached.view;
}
const key = JSON.stringify([this.path, identity, digest]);
let proof = pendingVerification.get(key);
Expand Down Expand Up @@ -420,7 +436,7 @@ export class FileAuthorityStore implements AuthorityStore {
// A failure after rename may already have published the new bytes.
// The next read must prove the actual file rather than reuse either
// the previous or attempted document.
verifiedDocument = null;
verifiedDocuments.delete(this.path);
return {
status: "ambiguous",
reason_code: "commit_outcome_unknown",
Expand Down
37 changes: 33 additions & 4 deletions loopx/control_plane/quota/cli_projection.py
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,9 @@ def _compact_nested_item_lists(
return compact


def _compact_agent_todo_summary(summary: dict[str, Any]) -> dict[str, Any]:
def _compact_agent_todo_summary(
summary: dict[str, Any], *, detail_command: str = QUOTA_CLI_TODO_SUMMARY_DETAIL_COMMAND
) -> dict[str, Any]:
compact: dict[str, Any] = {}
omitted_lanes: dict[str, int] = {}
for key, value in summary.items():
Expand Down Expand Up @@ -327,12 +329,14 @@ def _compact_agent_todo_summary(summary: dict[str, Any]) -> dict[str, Any]:
if lane != "current_agent_blocker_items" or summary.get(lane)
),
"omitted_lanes": omitted_lanes,
"full_detail_cold_path": QUOTA_CLI_TODO_SUMMARY_DETAIL_COMMAND,
"full_detail_cold_path": detail_command,
}
return compact


def _compact_user_todo_summary(summary: dict[str, Any]) -> dict[str, Any]:
def _compact_user_todo_summary(
summary: dict[str, Any], *, detail_command: str = QUOTA_CLI_USER_TODO_SUMMARY_DETAIL_COMMAND
) -> dict[str, Any]:
compact: dict[str, Any] = {}
omitted_lanes: dict[str, int] = {}
for key, value in summary.items():
Expand All @@ -359,7 +363,7 @@ def _compact_user_todo_summary(summary: dict[str, Any]) -> dict[str, Any]:
"schema_version": QUOTA_CLI_USER_TODO_SUMMARY_COMPACTION_SCHEMA_VERSION,
"retained_item_lanes": sorted(_RETAINED_USER_ITEM_LANES),
"omitted_lanes": omitted_lanes,
"full_detail_cold_path": QUOTA_CLI_USER_TODO_SUMMARY_DETAIL_COMMAND,
"full_detail_cold_path": detail_command,
}
return compact

Expand Down Expand Up @@ -790,3 +794,28 @@ def compact_quota_should_run_cli_payload(
return _promote_interaction_contract(
_promote_runtime_capability_reentry(compact)
)


def compact_quota_plan_cli_payload(
payload: dict[str, Any], *, detail_sections: frozenset[str] = frozenset()
) -> dict[str, Any]:
"""Bound read-only CLI summaries after complete planning; retain full opt-ins."""
if payload.get("mode") not in {"status", "plan"} or not isinstance(payload.get("groups"), dict):
return payload

def project_row(row: dict[str, Any]) -> dict[str, Any]:
result = dict(row)
for role, compact_summary in (("agent", _compact_agent_todo_summary), ("user", _compact_user_todo_summary)):
key = f"{role}_todos"
if f"{role}-todos" not in detail_sections and isinstance(row.get(key), dict):
result[key] = compact_summary(row[key], detail_command=(
f"quota {payload['mode']} --include-detail {role}-todos"
))
return result

result = dict(payload)
result["groups"] = {state: [project_row(row) for row in rows]
for state, rows in payload["groups"].items()}
if isinstance(payload.get("next_automatic_turn"), dict):
result["next_automatic_turn"] = project_row(payload["next_automatic_turn"])
return result
Loading
Loading