feat(coordination): fence NoKV authority publication on the workbench incarnation

Every NoKVAuthorityStore publication now names the workbench incarnation the
envelope was read from (`expected_workspace_incarnation_id`). NoKV 0.11.1
evaluates that fence atomically with the generation before any durable row or
object exists and refuses a stale incarnation with a typed exception, so a
workbench restored to a new incarnation between the read and the publish
refuses the write instead of accepting it at a restarted generation. The
JSON-lines helper maps that refusal to `failed/store_identity_mismatch`
(the same vocabulary the PostgreSQL service uses for a stale incarnation) and
keeps a refusal that names a different fence, or an untyped RuntimeError, on
the ambiguous path. Successful publications are still accepted only after the
current-incarnation readback required by RFC 6.2.

The helper pins NoKV SDK 0.11.1 / API 1 and admits only a wheel whose
`Client.publish_bytes` names the fence parameter and whose module exports
`WorkspaceIncarnationMismatch`; a 0.11.1-labelled wheel without that surface
is refused as `nokv_sdk_capability_mismatch` before any client is constructed,
and the request path rejects a publication without a valid fence before the
SDK call. The Stage 2A live probe gains two checks that publish the
generation-1 envelope with a stale fence and prove the typed refusal, the
unchanged generation and the unchanged workbench identity; the ladder row
`s2a.nokv_live_qualification` requires both checks and the 0.11.1 pin.

Tests: helper unit tests for the typed refusal, the fence validation and the
admission matrix; a fake SDK fixture with 0.11.1, 0.11.0 and 0.11.1-unfenced
shapes; transport tests through the real helper process for the refusal and
both admission rejections; store tests for the fence on every publication, the
refused incarnation race and the fence-ignoring owner; harness tests for the
new checks and a fence-ignoring backend; a pin consistency test across the
helper, ladder and probe.

Signed-off-by: wchwawa <wch19961116@gmail.com>

Author: wchwawa

examples/nokv-authority-store/README.md

@@ -17,14 +17,23 @@ This probe intentionally exercises the current Python SDK bridge because it is
 the available raw byte-CAS seam. It always starts this checkout's reviewed
 `NoKVJsonLinesTransport` and `nokv_jsonl_helper.py`; it has no fake, skip, or
 "unverified but successful" CLI path. The helper admits exactly NoKV SDK
-`0.11.0` / Python API `1`, and the successful report repeats both values.
+`0.11.1` / Python API `1` whose `Client.publish_bytes` names
+`expected_workspace_incarnation_id` and whose module exports
+`WorkspaceIncarnationMismatch`; the successful report repeats both version
+values. A `0.11.0` wheel is refused by the version pin (`invalid_config`), and a
+wheel labelled `0.11.1` without that publication surface is refused as
+`nokv_sdk_capability_mismatch` before any client is constructed.
 
 ## What it proves
 
 Against one **already existing** NoKV workbench, the probe starts three
 independent helper processes and verifies:
 
 - the selected tenant/goal path is initially absent and can be created;
+- a publication of the generation-1 envelope fenced on a stale workbench
+  incarnation is refused typed (`store_identity_mismatch`) before any row or
+  object exists: the stored generation stays 1 and the workbench identity is
+  unchanged (every LoopX publication carries the incarnation it was read from);
 - the stored path generation advances from 1 to 2 under exact generation CAS;
 - after the generation-2 CAS lands, an injected response loss is reconciled
   from the durable authority envelope and operation receipt rather than from
@@ -42,30 +51,34 @@ If the SDK, helper, workbench, backend, CAS, or independent readback cannot be
 proved, the process exits nonzero. The normal test suite uses deterministic
 fakes only to test this sequence and does **not** count as live evidence.
 
-The probe does not prove an atomic expected-incarnation publication fence,
-runtime shadow parity, a multi-Agent canary, authority promotion, HA, failover,
-restart recovery, capacity, or performance. NoKV generation can restart after
-workbench recreation, so the current adapter fails closed through authoritative
-post-write readback; preventing the stale-incarnation write itself requires a
-future provider primitive. The probe also does not create a workbench. A green
-run is Stage 2A single-node storage conformance evidence only.
+The probe proves the expected-incarnation publication fence only as far as a
+single live owner can show it: a stale fence is refused typed and leaves the
+generation untouched. It does not restore or recreate the workbench (NoKV
+exposes no client-side retire verb), so the incarnation rotation itself is
+covered by NoKV's own executor tests, not by this probe. Success is still
+accepted only after authoritative post-write readback in the current
+incarnation: the fence removes the stale write, not the readback obligation.
+The probe does not prove runtime shadow parity, a multi-Agent canary, authority
+promotion, HA, failover, restart recovery, capacity, or performance, and it does
+not create a workbench. A green run is Stage 2A single-node storage conformance
+evidence only.
 
 ## Inputs
 
 Use a current NoKV Python environment. Keep the client configuration in an
 ignored local file; do not commit credentials. The helper accepts three routing
 kinds and passes each to the matching `RoutingConfig` constructor of the
-installed SDK: `etcd` and `static` (the 0.11.0 release wheel) and `seeds`
+installed SDK: `etcd` and `static` (the 0.11.x release wheels) and `seeds`
 (`{"kind": "seeds", "endpoints": ["IP:PORT", ...]}`, the NoKV
 metadata-runtimes line, which names serving owners directly and drops the etcd
 constructor). Static routing is valid for a single-node NoKV deployment; etcd
 is not required by this probe. A routing kind the installed wheel cannot build
 fails the open handshake with `nokv_sdk_capability_mismatch` before any client
-is constructed, so a seeds configuration against a 0.11.0 wheel (or an etcd
+is constructed, so a seeds configuration against a 0.11.1 wheel (or an etcd
 configuration against a metadata-runtimes wheel) is reported as the wrong
 wheel, not as an outage. The `ready` handshake echoes `nokv_protocol_schema`:
 the SDK's `WORKSPACE_PROTOCOL_SCHEMA` when the wheel exports one, otherwise
-`null` (the 0.11.0 release does not). The following shape is illustrative:
+`null` (the 0.11.x releases do not). The following shape is illustrative:
 
 ```json
 {
@@ -136,7 +149,7 @@ unfenced, pre-existing, or unreadable state exits nonzero with a compact JSON
 reason; provider stderr, endpoints, credentials, and raw SDK errors are not
 copied into that result. A successful JSON report includes
 `"qualification_scope":"stage_2a_single_node_store_conformance"`,
-`"nokv_sdk_version":"0.11.0"`, and `"nokv_api_version":1`. The two version
+`"nokv_sdk_version":"0.11.1"`, and `"nokv_api_version":1`. The two version
 fields are the helper's admission constants: the helper refuses to open a client
 for any other SDK version or API version, so a successful report implies them,
 but they are not values read back from the NoKV server. The report is Stage

examples/nokv-authority-store/live-qualification.ts

@@ -30,7 +30,7 @@ import {
 
 const REPORT_SCHEMA = "loopx_nokv_authority_live_qualification_v0";
 export const QUALIFICATION_SCOPE = "stage_2a_single_node_store_conformance";
-export const QUALIFIED_NOKV_SDK_VERSION = "0.11.0";
+export const QUALIFIED_NOKV_SDK_VERSION = "0.11.1";
 export const QUALIFIED_NOKV_API_VERSION = 1;
 const REPOSITORY_HELPER = fileURLToPath(
   new URL("../../loopx/control_plane/coordination/nokv_jsonl_helper.py", import.meta.url),
@@ -266,6 +266,16 @@ function applied(
   return result;
 }
 
+/** A 32-hex incarnation that differs from `current` in its first digit. */
+function staleIncarnation(current: string): string {
+  return `${current.startsWith("0") ? "1" : "0"}${current.slice(1)}`;
+}
+
+/** A fresh lower-layer publication identity; the probe never reuses one. */
+function freshPhysicalIdentity(): string {
+  return randomUUID().replaceAll("-", "");
+}
+
 async function rawGeneration(
   transport: NoKVBlobTransport,
   store: NoKVAuthorityStore,
@@ -361,6 +371,47 @@ export async function exerciseQualificationSequence(
     await rawGeneration(firstTransport, first, 1, "create_generation_failed");
     passed("create_generation_one");
 
+    // A write prepared against one workbench incarnation must not land after
+    // the workbench is restored to another. Send the generation-1 envelope back
+    // through the raw transport with a fence naming a different incarnation:
+    // NoKV must refuse it typed, before any row or object exists, and the
+    // stored generation and the workbench identity must both be unchanged.
+    const boundIdentity = firstIdentity.status === "available"
+      ? firstIdentity.store_identity
+      : fail("workbench_identity_failed", "workbench identity was not available");
+    const currentEnvelope = await firstTransport.readBlob(first.workbench, first.path);
+    expect(
+      currentEnvelope.status === "loaded" && currentEnvelope.generation === 1,
+      "stale_incarnation_fence_probe_failed",
+      "the generation-1 envelope could not be read for the fence probe",
+    );
+    const staleFence = await firstTransport.casPublishBlob({
+      workbench: first.workbench,
+      path: first.path,
+      expected_generation: 1,
+      expected_workspace_incarnation_id: staleIncarnation(
+        boundIdentity.slice(`nokv:${workbench}:`.length),
+      ),
+      bytes: currentEnvelope.bytes,
+      operation_id: freshPhysicalIdentity(),
+      artifact_revision_id: freshPhysicalIdentity(),
+    });
+    expect(
+      staleFence.status === "failed" && staleFence.reason_code === "store_identity_mismatch",
+      "stale_incarnation_fence_not_enforced",
+      "NoKV did not refuse a publication fenced on a stale workbench incarnation",
+    );
+    passed("stale_incarnation_fence_rejected");
+    const identityAfterFence = await first.storeIdentity();
+    expect(
+      identityAfterFence.status === "available" &&
+        identityAfterFence.store_identity === boundIdentity,
+      "stale_incarnation_fence_probe_failed",
+      "the workbench incarnation changed during the fence probe",
+    );
+    await rawGeneration(firstTransport, first, 1, "stale_incarnation_fence_wrote");
+    passed("stale_incarnation_fence_left_generation_unchanged");
+
     const advanced = applied(
       await second.commitAuthority(
         commit(created.provider_revision, runId, operationIds.advance, 2, "advance"),

examples/shared-goal-authority-e2e/README.md

@@ -35,7 +35,7 @@ suites rather than in the pytest shards.
 | `s0.file_matrix_twelve_rows` | 0 | store_direct | deterministic | `examples/nokv-shadow-provider/live_e2e.py` reports exactly the twelve known file-provider scenario rows, all true |
 | `s0.nokv_live_matrix` | 0 | store_direct | env:nokv_legacy | the same twelve rows plus `restored_lineage_fails_closed` are true on a live NoKV stack and file/NoKV outcomes are identical |
 | `s1.cli_document_decodes_through_ts_store` | 1 | real_cli | deterministic | three CLI writes (`todo add`, `task-lease acquire`, `todo update`) read back through `FileAuthorityStore`: `loadAuthority` loaded at cursor `3`, paged `scanCommitted` yields the three `observation_id`s in order, `readReceipt` finds the first |
-| `s2a.nokv_live_qualification` | 2a | store_direct | env:nokv_authority | runs the merged `examples/nokv-authority-store/live-qualification.ts --execute-live` against an existing workbench with a fresh tenant/goal pair; requires `ok=true`, the single-node store-conformance scope, every check `passed`, NoKV SDK `0.11.0` / API `1`, and no promotion or availability claim; evidence carries check ids, counts, and config and workbench digest prefixes, never a configuration value or the workbench name |
+| `s2a.nokv_live_qualification` | 2a | store_direct | env:nokv_authority | runs the merged `examples/nokv-authority-store/live-qualification.ts --execute-live` against an existing workbench with a fresh tenant/goal pair; requires `ok=true`, the single-node store-conformance scope, every check `passed`, NoKV SDK `0.11.1` / API `1`, the two stale-incarnation fence checks (`stale_incarnation_fence_rejected`, `stale_incarnation_fence_left_generation_unchanged`), and no promotion or availability claim; evidence carries check ids, counts, and config and workbench digest prefixes, never a configuration value or the workbench name |
 | `s2b.postgresql_conformance_live` | 2b | store_direct | env:postgresql | `postgresql_authority_store.integration.test.ts` under node's TAP reporter: `# pass >= 9`, `# fail 0`, `# skipped 0` |
 | `s2c1.configure_enable_disable_roundtrip` | 2c1 | real_cli | deterministic | `configure-goal` preview does not write, enable writes, captured observations for a todo and a lease, read-back summary `enabled/file_one_way`, disable writes and later writes neither observe nor touch candidate bytes |
 | `s2c1.every_writer_family_captures` | 2c1 | real_cli | deterministic | handoff-mode set, todo add/update/complete/supersede/archive-completed, task-lease acquire/renew/transfer each carry `outcome in {captured, replayed, ambiguous_reconciled}`, `primary_writeback_preserved=true`, `provider_to_local_writes=false`, `candidate_read_for_decision=false`; an idempotent re-acquire carries no `authority_shadow`; candidate `cursor == captured count`, operation ids equal observation ids, no time-active lease in the head, head todos equal `todo list` |
@@ -86,7 +86,7 @@ TypeScript store read.
 | `deterministic` | none (needs `node` on `PATH` for the CLI's TypeScript runtime and the read-back probe) | `node_missing` when the probe cannot run |
 | `env:postgresql` | `LOOPX_TEST_POSTGRES_URL` plus `node_modules/pg` (`npm ci`) | `postgres_url_missing`, `pg_dependency_missing`, `node_missing` |
 | `env:nokv_legacy` | `NOKV_COORDINATION_LIVE=1` and `NOKV_ETCD`, `NOKV_ETCD_PREFIX`, `NOKV_ROOT_ID`, `NOKV_BUCKET`, `NOKV_OBJECT_ENDPOINT`, `NOKV_OBJECT_ROOT`, `NOKV_OBJECT_KEY`, `NOKV_OBJECT_SECRET`; the `nokv` SDK importable | `nokv_live_env_missing`, `nokv_coordination_live_not_enabled`, `nokv_sdk_missing` |
-| `env:nokv_authority` | `LOOPX_NOKV_AUTHORITY_LIVE=1` (the probe writes durable test data), `LOOPX_NOKV_AUTHORITY_CONFIG_JSON` (absolute path to the ignored NoKV client configuration), `LOOPX_NOKV_AUTHORITY_PYTHON` (absolute path to the Python executable that resolves NoKV SDK 0.11.0), `LOOPX_NOKV_AUTHORITY_WORKBENCH` (an existing workbench); `node` on `PATH` | `nokv_authority_env_missing`, `loopx_nokv_authority_live_not_enabled`, `nokv_authority_config_missing`, `nokv_authority_python_missing`, `node_missing` |
+| `env:nokv_authority` | `LOOPX_NOKV_AUTHORITY_LIVE=1` (the probe writes durable test data), `LOOPX_NOKV_AUTHORITY_CONFIG_JSON` (absolute path to the ignored NoKV client configuration), `LOOPX_NOKV_AUTHORITY_PYTHON` (absolute path to the Python executable that resolves NoKV SDK 0.11.1), `LOOPX_NOKV_AUTHORITY_WORKBENCH` (an existing workbench); `node` on `PATH` | `nokv_authority_env_missing`, `loopx_nokv_authority_live_not_enabled`, `nokv_authority_config_missing`, `nokv_authority_python_missing`, `node_missing` |
 
 POSIX-only rows report `unverified/posix_only` on Windows.
 

loopx/control_plane/coordination/nokv_authority_store.ts

@@ -54,6 +54,12 @@ export interface NoKVBlobCasRequest {
   bytes: Uint8Array;
   operation_id: string;
   artifact_revision_id: string;
+  /**
+   * The workbench incarnation this publication is bound to. NoKV evaluates it
+   * atomically with `expected_generation` before any durable row or object
+   * exists; a stale value is refused as `failed/store_identity_mismatch`.
+   */
+  expected_workspace_incarnation_id: string;
 }
 
 export type NoKVBlobCasResult =
@@ -186,6 +192,11 @@ function readFailure(error: unknown): AuthorityStoreReadFailure {
   };
 }
 
+/** The incarnation half of a validated `nokv:{workbench}:{incarnation}` identity. */
+function boundIncarnation(storeIdentity: string, workbench: string): string {
+  return storeIdentity.slice(`nokv:${workbench}:`.length);
+}
+
 function validStoreIdentity(value: string, workbench: string): boolean {
   const prefix = `nokv:${workbench}:`;
   return value.startsWith(prefix) && HEX_128_PATTERN.test(value.slice(prefix.length));
@@ -402,13 +413,18 @@ export class NoKVAuthorityStore implements AuthorityStore {
     // attempt. Keep the LoopX operation id stable in the authority envelope,
     // while giving each physical retry a fresh pair of lower-layer ids. A
     // response-lost success is still settled only by reading that envelope.
+    // The request also names the incarnation the envelope was read from, so a
+    // workbench restored to a new incarnation between this read and the
+    // publish refuses the write instead of accepting it at a restarted
+    // generation.
     const attemptNonce = randomUUID();
     let result: NoKVBlobCasResult;
     try {
       result = await this.transport.casPublishBlob({
         workbench: this.workbench,
         path: this.path,
         expected_generation: expectedGeneration,
+        expected_workspace_incarnation_id: boundIncarnation(current.identity, this.workbench),
         bytes: payload,
         operation_id: physicalAttemptIdentity(
           "operation",
@@ -433,11 +449,11 @@ export class NoKVAuthorityStore implements AuthorityStore {
       };
     }
     if (result.status === "applied" && result.generation === generation) {
-      // Generation is not a workbench-incarnation fence: NoKV may restart it
-      // after remove/recreate. Never expose success until a fresh read proves
-      // this exact transaction in the current incarnation. Preventing the
-      // stale-incarnation write itself still requires an atomic provider
-      // primitive that accepts the expected incarnation.
+      // The incarnation fence removes the stale write, not the readback
+      // obligation: success is exposed only after a fresh read proves this
+      // exact transaction in the current incarnation (RFC §6.2), so an owner
+      // that ignored the fence still cannot make a restarted generation look
+      // like a LoopX commit.
       return await this.settleCommitFromReadback(
         normalized.expected_provider_revision,
         transaction,