feat(coordination): fence NoKV authority publication on the workbench incarnation (NoKV 0.11.1) - #4774
Conversation
… incarnation Every NoKVAuthorityStore publication now names the workbench incarnation the envelope was read from (`expected_workspace_incarnation_id`). NoKV 0.11.1 evaluates that fence atomically with the generation before any durable row or object exists and refuses a stale incarnation with a typed exception, so a workbench restored to a new incarnation between the read and the publish refuses the write instead of accepting it at a restarted generation. The JSON-lines helper maps that refusal to `failed/store_identity_mismatch` (the same vocabulary the PostgreSQL service uses for a stale incarnation) and keeps a refusal that names a different fence, or an untyped RuntimeError, on the ambiguous path. Successful publications are still accepted only after the current-incarnation readback required by RFC 6.2. The helper pins NoKV SDK 0.11.1 / API 1 and admits only a wheel whose `Client.publish_bytes` names the fence parameter and whose module exports `WorkspaceIncarnationMismatch`; a 0.11.1-labelled wheel without that surface is refused as `nokv_sdk_capability_mismatch` before any client is constructed, and the request path rejects a publication without a valid fence before the SDK call. The Stage 2A live probe gains two checks that publish the generation-1 envelope with a stale fence and prove the typed refusal, the unchanged generation and the unchanged workbench identity; the ladder row `s2a.nokv_live_qualification` requires both checks and the 0.11.1 pin. Tests: helper unit tests for the typed refusal, the fence validation and the admission matrix; a fake SDK fixture with 0.11.1, 0.11.0 and 0.11.1-unfenced shapes; transport tests through the real helper process for the refusal and both admission rejections; store tests for the fence on every publication, the refused incarnation race and the fence-ignoring owner; harness tests for the new checks and a fence-ignoring backend; a pin consistency test across the helper, ladder and probe. Signed-off-by: wchwawa <wch19961116@gmail.com>
huangruiteng
left a comment
huangruiteng
left a comment
There was a problem hiding this comment.
动机
这次改动解决的是 NoKV Stage 2A 里最后一个明确的写入竞态：LoopX 在 workbench incarnation A 读取 authority envelope 后，如果 workbench 在 publish 前被重建为 incarnation B，NoKV 的 generation 可能从头开始。旧实现的 current-incarnation readback 能避免把这次写入误报为成功，却不能阻止旧 incarnation 的字节先落到新 workbench。
我核对了共享 authority RFC、#4069 的职责边界以及 NoKV 上游 #514 / v0.11.1 的实现证据。这个 PR 的边界是合理的独立增量：把上游原子 incarnation fence 接到 LoopX 既有 NoKV adapter 和 Stage 2A qualification；不修改 provider profile hold，不激活运行时 provider，也不声称完成 promotion、HA 或多 owner 验证。
改动思路
权责分层保持清楚：TypeScript NoKVAuthorityStore 继续负责 authority document、retained receipts 和 readback settlement；Python helper 只做 SDK surface admission、输入校验和 typed exception 适配；NoKV 0.11.1 owner 在持久化任何 operation row、artifact revision 或 object 之前，原子校验 expected_workspace_incarnation_id 与 generation。
正向路径是：从已验证的 nokv:{workbench}:{incarnation} identity 派生 fence，连同 generation 一起 publish；provider 接受后仍必须重新读取 current-incarnation envelope，并看到本次 transaction 才返回 applied。负向路径是：A→B 的竞态触发 WorkspaceIncarnationMismatch，helper 仅在 exception type 正确且 .expected 等于本请求 fence 时映射为 failed/store_identity_mismatch；不匹配的 exception 或普通 RuntimeError 继续保持 ambiguous，避免把 response loss 错判成无副作用拒绝。
具体改动
- 内部
NoKVBlobCasRequest新增必填 incarnation fence；commitAuthority在每次 physical publish（包括逻辑重试）上都携带从当前已验证 identity 派生的 fence，同时保持 fresh physical operation/revision ids。 - JSON-lines transport 透传该字段；helper 把 SDK pin 提升到 0.11.1，并要求
Client.publish_bytes显式声明该参数、模块导出真正的WorkspaceIncarnationMismatch。缺字段、格式错误、旧 wheel 或伪装成 0.11.1 但缺 capability 的 wheel 都 fail closed。 - Stage 2A live qualification 增加 stale-fence write：要求 typed refusal、workbench identity 不变、generation 仍为 1；E2E ladder 必须看到两个对应 passed check 才接受报告。
- 文档准确区分了“阻止 stale write”和“仍需 readback”，并继续明确 live probe 不证明 promotion、HA、failover、多 owner 或性能。
关键代码讲解
NoKVAuthorityStore.commitAuthority：fence 与 envelope 来自同一次已验证 identity；provider 返回applied也不能绕过 current-incarnation readback。这个位置继续是 LoopX authority settlement 的唯一 owner。publish_incarnation_fence_supported：同时校验显式参数与 typed refusal class。只接受**kwargs或只有其中一半会造成 silent downgrade，因此在 client 构造前拒绝是必要的。_cas_publish_blob：请求 fence 先做 32-hex 校验；捕获 typed mismatch 后还比对.expected，避免把属于另一个请求的异常当成本请求无副作用的证明。exerciseQualificationSequence：在 generation 1 后直接对同一路径发送错误 incarnation 的 raw publish，证明拒绝和无写入，再继续原有 response-loss、CAS contention 和独立 readback 序列。authority_e2e_ladder.py：把两个 fence check id 和 SDK 0.11.1 pin 纳入可机器验证的 qualification receipt，而不是仅依赖文档描述。
对主干的风险
最强反例是“某条 retry 漏传 fence，或 helper 把无关 RuntimeError 当成 typed mismatch”：前者重新打开 stale write 窗口，后者会错误声称没有持久化副作用。本 head 通过必填 TypeScript request、每次 attempt 的传播、SDK admission、.expected 精确比对，以及 owner-ignore readback guard 覆盖了这两类风险。
我用同一个 incarnation-rotating transport 对 pre-change parent 与本 head 做了对照：base 返回 provider_protocol_violation，但已经 wrote=true, generation=1, request_has_fence=false；本 head 返回 store_identity_mismatch，且 wrote=false, generation=null, request_has_fence=true。这证明改变的不只是 reason code，而是持久化副作用本身。
本地验证结果：
- Python helper + shared authority E2E：72 passed，8 个 external-provider 环境门禁用例明确 skipped；
- TypeScript store/transport/qualification（含注册的共享 authority conformance）：343 passed，0 failed，0 skipped；
- Ruff、control-plane TypeScript typecheck、
git diff --check均通过； - 独立安装并检查已发布的 NoKV v0.11.1 wheel：版本/API、
publish_bytes参数、typed exception 与.expected均符合，当前 helper admission predicate 接受真实nokv.Client； - 与当前
origin/main的 merge-tree 无冲突，main 在 packet base 后未修改本 PR 的 13 个路径。
剩余风险是我没有可用的 live NoKV owner/workbench，未亲自重跑写入型 Stage 2A probe。这个缺口没有被隐藏：真实 wheel surface 和 NoKV 上游原子合同已经独立核对，LoopX 的真实 helper process/authority conformance 已执行，而 live proof 仍是 profile promotion 前的显式环境门禁。本 PR 本身不移除该 hold，因此不阻塞合入这段 qualification implementation。
语义与 CI 对齐
该 PR 扩展现有 NoKV candidate vocabulary：复用上游 expected_workspace_incarnation_id / WorkspaceIncarnationMismatch 和 LoopX 已有 store_identity_mismatch，没有创建第二套 authority 或 actor lifecycle 术语。helper、probe、ladder、fake SDK、测试和 README 全部对齐到 NoKV 0.11.1 / API 1 及同一对 check ids；provider profile/RFC hold 保持不变，避免“实现存在”被误解释为“已获 promotion authority”。按本 packet 契约没有查询或等待远端 CI，结论基于 exact-head 的 repository-native local validation。
我的整体评价
我没有发现 blocking finding。实现把原本 write-then-detect 的竞态收紧为 provider-side no-write refusal，同时保留 readback、ambiguous outcome 和 typed authority ownership 的既有边界；机制规模与风险相称，也没有引入新的 persisted state、runtime switch 或 speculative framework。
未来相关的小步优化里，最值得坚持的是让 TypeScript 继续拥有 authority state machine、Python 只适配 provider contract；本 PR 已经遵守这个边界，不需要在本次扩大重构。
结论：批准 exact head 9cffc78be344454f04a4ed65ee3f906af9b41c28。
English verdict: APPROVE - head 9cffc78
2 participants
Goal And Delivered Outcome
publish_bytes; preventing a write after a concurrent remove/recreate remains an explicit provider-contract hold" (profile holdatomic_workspace_incarnation_publication_fence). NoKV 0.11.1 (feat: fence artifact publication on the expected workspace incarnation NoKV-Lab/NoKV#514, closes feat(publish): atomic expected workspace incarnation fence on publish_bytes NoKV-Lab/NoKV#508, released asv0.11.1) addsexpected_workspace_incarnation_idtoClient.publish_bytes/publish_file, evaluated by the owner atomically withexpected_generationbefore any durable row or object exists, and refuses a stale incarnation with the typednokv.WorkspaceIncarnationMismatch. This PR consumes that primitive at the NoKV plane, within the review scope accepted in Scoped review invitations and module stewardship #4069 (NoKV-plane evidence only; no promotion, provider activation, or normative RFC edits).failed/provider_protocol_violation, "lineage mismatch"), so a restarted generation could already hold LoopX bytes. After, everyNoKVAuthorityStorepublication names the incarnation the envelope was read from; the owner refuses a stale one before it exists and the helper reportsfailed/store_identity_mismatch(the vocabulary the PostgreSQL service already uses for a stale incarnation); the stored generation and the workbench identity are unchanged, and the current-incarnation readback obligation of RFC §6.2 is kept. Proved by theunitstore rows (fence on every publication, refused incarnation race, fence-ignoring owner still fails closed), thereal_entrypointprobe checksstale_incarnation_fence_rejected/stale_incarnation_fence_left_generation_unchanged, and thereal_backendladder rows below.main.Scope And Continuation
nokv_jsonl_helper.pypins NoKV SDK0.11.1/ API1, admits only a wheel whoseClient.publish_bytesnamesexpected_workspace_incarnation_idand whose module exportsWorkspaceIncarnationMismatch(a 0.11.1-labelled wheel without that surface is refused asnokv_sdk_capability_mismatchbefore any client is constructed), requires a valid fence on everycas_publish_blobbefore the SDK call, and maps the typed refusal tofailed/store_identity_mismatchwhile a refusal naming a different fence stays on the ambiguousprovider_protocol_violationpath; (2)NoKVBlobCasRequest.expected_workspace_incarnation_idis required and the store fills it from the identity the envelope was read under, the JSON-lines transport forwards it; (3) the Stage 2A live probe publishes the generation-1 envelope with a stale fence and proves the typed refusal, the unchanged generation and the unchanged workbench identity; (4) ladder rows2a.nokv_live_qualificationrequires both checks and the 0.11.1 pin and reportsincarnation_fence_checks; (5) fixtures, tests and the two READMEs. Remaining, owner-side and normative (deliberately not edited here, see the proposal at the end): the profile hold and the RFC sentences that describe the hold and the 0.11.0 pin. Not proven by this PR: the incarnation rotation itself (NoKV exposes no client-side retire/recreate verb; the rotation is covered by NoKV's executor tests in feat: fence artifact publication on the expected workspace incarnation NoKV-Lab/NoKV#514, and the live probe can only show the refusal of a stale fence against the current incarnation), restart/restore recovery, capacity ([Design] Bounded NoKV authority layout: sealed 64-commit segments and an operation index replacing the retain-all 16 MiB envelope (owner decisions Q5/capacity vocabulary) #4727), availability/HA.Validation
9cffc78be344454f04a4ed65ee3f906af9b41c28staticpassedpython -m ruff checkon the changed Python files;python -m mypy(pyproject file list, 22 files, no issues);npm run typecheck:control-planeclean.unitpassedtests/test_nokv_jsonl_helper.py: 48 passed. Covers the typed refusal (store_identity_mismatch), a refusal naming another fence (ambiguousprovider_protocol_violation), an untypedRuntimeErrorstaying ambiguous, seven invalid/missing fence values rejected before the SDK call, and the admission matrix (parameter missing, parameter only through**kwargs, typed refusal missing or not a class, the 0.11.0 shape under a 0.11.1 label, the 0.11.0 version itself).unitpassedtests/control_plane/test_shared_goal_authority_e2e.py: 72 passed, 8 skipped (environment-gated live rows), including the new consistency test that pins the SDK version and the two fence check ids identically across the helper, the ladder and the probe.unitpassedtests/control_plane_ts/nokv_authority_store.test.ts,nokv_jsonl_transport.test.ts,nokv_stage2a_qualification_harness.test.ts: 343 pass, 0 fail. The transport tests run the real helper process against the fake SDK fixture in its0.11.1,0.11.0and0.11.1-unfencedshapes (typed refusal with unchanged generation; version-pin refusal; capability refusal). The store tests cover the fence on every physical retry, the refused incarnation race with unchanged bytes/generation and lineage-fenced readback afterwards, and an owner that ignores the fence still failing closed through readback. Both NoKVregisterAuthorityStoreConformancearms (fake transport and JSON-lines process) pass.unitpassedtests/control_plane_ts/*.test.tson Node 26 (the runtime of CI'stest:control-planejob): 1990 tests, 1982 pass, 1 skipped, 7 fail. All 7 failures are pre-existing and reproduce identically on the unmodified base commita20e71395with the same runtime (two PostgreSQL integration files that need a live database,goal_acceptance_authority,goal_acceptance_runtime,team_plan, and two single assertions inmonitor_successor/quota_monitor_poll_committhat Node 26.4 rejects with "The message argument must be one of type string or function"); none of them import the NoKV plane. Every NoKV-related file passes on both Node 24 and Node 26.real_entrypointpassedexamples/nokv-authority-store/live-qualification.ts --execute-livethrough the ladder: 15 checks passed, includingstale_incarnation_fence_rejectedandstale_incarnation_fence_left_generation_unchanged; reportnokv_sdk_version0.11.1, final generation 3.real_backendpasseds0.nokv_live_matrixands2a.nokv_live_qualificationat the tested revision, clean tree, against a local single-node NoKV 0.11.1 owner (etcd control path, S3-compatible object store, isolated dev stack) with the releasednokv==0.11.1macOS arm64 wheel installed from the GitHub release and checksum-verified against itsSHA256SUMS: 2 pass, 0 fail, privacy scan 0 violations.regression_paritypassednokv==0.11.0wheel:s0passes (the matrix does not pin the SDK),s2afails typed at helper admission (probenokv_transport_protocol_failed, helper reason "must be version 0.11.1"), no write reached the owner. Failing-before shape: the harness test with a fence-ignoring backend failsstale_incarnation_fence_not_enforcedand shows the unfenced stale write had moved the envelope to generation 2 without a LoopX commit, which is exactly what the fence removes.unitnot_runtests/control_plane_ts/authority_provider_parity.test.tsSQLite arms cannot run on the local Node 24 runtime (SQLite runtime not qualified; identical failure on unmodifiedmain); the NoKV arm is covered by the store tests above and the file is included in the Node 26 full-suite row.stage2c-suite(runs only onkind == "full"or theci:fulllabel; the live rows above were run locally because of that).See validation disclosure guidance.
Frontend / Visual Evidence
Type of Change
LoopX Area
Technical Direction
loopx-overall-roadmap-v0). Candidate evidence only: no promotion default, no provider activation, D1–D3 unchanged.Shared-authority RFC fixture impact
store_identity_mismatchfor a refused stale fence; the persisted envelope schemaloopx_nokv_authority_store_v0, cursor/receipt semantics and the readback rule are unchanged.registerAuthorityStoreConformanceregistrations) pass; File, SQLite and PostgreSQL arms are untouched by this diff.Boundary Checklist
.loopx/,.codex/goals/, and liveACTIVE_GOAL_STATE.md).none.Signed-off-bytrailer (git commit -s).Proposal for the RFC owner (normative, not changed in this PR)
If this PR is accepted, the following owner-side edits follow from it; I am not making them because they are normative:
loopx/control_plane/coordination/authority_store.ts, NoKV profilequalification_holds: removeatomic_workspace_incarnation_publication_fence(onlycapacity_and_receipt_retentionis pinned byauthority_store.test.ts). The other holds stay.0.11.1, and the sentence "NoKV's current Python API does not atomically bind an expected workspace incarnation intopublish_bytes; preventing a write after a concurrent remove/recreate remains an explicit provider-contract hold" → NoKV 0.11.1 binds the expected incarnation atomically with the generation before any durable row or object exists; the LoopX store names it on every publication and the live probe proves the typed refusal; the rotation itself is exercised by NoKV's executor tests, and LoopX still accepts success only after current-incarnation readback.7bb3ffd6…(release 0.11.0 …)" → NoKV590d3a4bdc…(release 0.11.1, Python API 1, Holt 0.8.6), and the Stage 2A row description "NoKV SDK0.11.0/ API1" →0.11.1plus the two fence checks (the ladder README in this PR already says so).store_identity_mismatchfor the refused fence; if the owner prefers a NoKV-specific code, it is one string in the helper and the probe.