feat(a2a): GPU lease protocol over the coordination bus (#893)

.gitignore

@@ -203,3 +203,11 @@ data/*.sqlite3-wal
 data/apps/
 data/.install_id
 data/store_popularity.json
+
+# Registry signing-key lock sidecar. Added 2026-09-12: `data/*.pem` (above)
+# covers the key itself, but not the 0-byte `*.pem.lock` flock sidecar the
+# registry key writer creates beside it - so it rode into PR #2988 on a stray
+# `git add`, the same accident the note above describes. Enumerated, not a
+# blanket data/ rule, for the reason given there.
+# Verified against every tracked file on origin/dev: 0 become ignored.
+data/*.pem.lock

changelog.d/taos-893-a2a-gpu-lease.md

@@ -0,0 +1,43 @@
+### Added
+
+- Shared-GPU coordination over the A2A bus (taOS #893). Agents sharing one card
+  can `GET /api/a2a/gpu/check` (folds the channel's open `[GPU CLAIM]`/
+  `[GPU RELEASE]` messages into the claims still held and subtracts them from
+  the node's live free VRAM), then `POST /api/a2a/gpu/claim` for an
+  admission-checked claim that both registers a real cluster lease (TTL, kept
+  alive with `POST /api/a2a/gpu/renew`) and posts the `[GPU CLAIM]` line peers
+  read; `POST /api/a2a/gpu/release` frees it and `POST /api/a2a/gpu/request`
+  asks for a window when blocked. The cluster lease applies to nodes the
+  controller knows as cluster workers; a node it does not know is coordinated
+  over the bus alone (admission-checked and posted, with no local TTL).
+  Claiming refuses (409) on another holder's
+  open claim or insufficient VRAM, and CHECK/CLAIM return 503 rather than
+  reporting a node free when the bus cannot be read, so two agents can no
+  longer silently co-load past the card's VRAM. An agent posts as its own
+  registry identity (scope `a2a_receive` to check, `a2a_send` to act) and the
+  node label resolves to a cluster worker's heartbeat VRAM or the controller's
+  own shared VRAM ledger.
+- Claims carry their own expiry. `POST /api/a2a/gpu/claim` publishes
+  `expires=<unix ts>` on the `[GPU CLAIM]` line (the backing cluster lease's
+  expiry, or the requested TTL for a bus-only node), rounded up so a published
+  expiry can never precede the lease it describes, and `POST /api/a2a/gpu/renew`
+  reposts that line as it extends the lease — on the channel the claim was made
+  on, since the channel is an input to the claim and never to its renewal. If the
+  repost fails the local extension is rolled back rather than leaving a lease
+  that peers have already seen lapse. The fold drops a claim whose published
+  expiry has passed, so a holder that crashed or stopped keeping alive no longer
+  blocks the shared card until its claim ages out of the fold window. A claim
+  posted by hand without an `expires=` is unchanged: it has no time-based
+  expiry, so only a RELEASE closes it - though it is still limited by the fold
+  window (the newest 500 messages), so a long-lived one is reposted
+  periodically.
+
+### Fixed
+
+- Freeing another holder's GPU lease by explicit `lease_id` (the operator
+  override, `POST /api/a2a/gpu/release`) posts the `[GPU RELEASE]` line as the
+  **freed holder** rather than as the operator. A claim is keyed on its bus
+  author, so the old line cleared nothing: the local lease was gone while every
+  peer's fold still read the node as claimed, blocking a GPU that was actually
+  free. The response now distinguishes `holder` (who acted) from
+  `released_holder` (whose claim the line closes).

docs/agent-coordination.md

@@ -351,6 +351,112 @@ Read through the controller with your own registry token, not the raw bus port:
 If the bus is silent, check `channel_known` and your cursor before concluding nobody is
 talking. A read that returns `200` with nothing is the failure mode that looks like peace.
 
+## Shared-GPU leases (`/api/a2a/gpu/*`)
+
+Two agents on one host share a physical GPU and must not silently co-load past
+its VRAM (taOS #893). Coordinate over the bus with a one-line text protocol, and
+use the controller's endpoints so the protocol is admission-checked and backed by
+a real lease:
+
+```text
+[GPU CLAIM] node=<host> holder=@you vram=~9.4gb reason=... eta=... expires=<unix ts>
+[GPU RELEASE] node=<host> holder=@you
+[GPU REQUEST] node=<host> need=~6gb
+```
+
+| Method | Path | Scope | Purpose |
+|--------|------|-------|---------|
+| GET  | `/api/a2a/gpu/check`   | `a2a_receive` | Fold the channel's open claims + the node's live VRAM and answer "may I load?" |
+| POST | `/api/a2a/gpu/claim`   | `a2a_send`    | Admission-checked claim: cluster lease (TTL) + `[GPU CLAIM]` post |
+| POST | `/api/a2a/gpu/release` | `a2a_send`    | Release the lease + `[GPU RELEASE]` post |
+| POST | `/api/a2a/gpu/request` | `a2a_send`    | Post `[GPU REQUEST]` when blocked |
+| POST | `/api/a2a/gpu/renew`   | `a2a_send`    | Keep-alive: extend a lease TTL |
+
+Do not post a claim line directly to the bus and skip `/claim`: only the endpoint
+checks admission (another holder's claim, the node's free VRAM, and the cluster's
+own lease table) before the line is posted. A line posted by hand is recorded but
+enforces nothing.
+
+The **local** half of a claim — the cluster lease with its TTL, renewal and
+rollback — applies to nodes this controller knows as cluster workers. A node that
+resolves to no worker is coordinated over the bus alone: CLAIM still admission-
+checks the channel and posts the line, but there is no local reservation to
+expire, so the bus-side `[GPU RELEASE]` is the only thing that frees it. The
+endpoint reports a node it cannot measure as `vram_verified: false` rather than
+as free.
+
+Rules that matter when you use it:
+
+- **CHECK before load, always.** It folds the channel's `[GPU CLAIM]`/`[GPU
+  RELEASE]` history into the claims still open, folds in this controller's own
+  GPU leases (which the bus never shows), and subtracts both from the node's
+  live free VRAM. A node claimed by ANY other holder is blocked even if the card
+  looks free, because "claimed" means a load is in flight.
+- **Holder identity is the bus author, not the `holder=` text.** The body is
+  caller-controlled; `from` is what the bus authenticated (see *Posting to the
+  coordination bus*). The `holder=` field is a readable label for humans.
+- **An agent always acts as itself.** `from` is the agent's registry canonical
+  id, the body's `holder=` is its registry handle, and the caller's own registry
+  JWT is forwarded to the bus exactly as on `/api/a2a/bus/send`. A `holder`
+  field in the request body is ignored for agent callers.
+- **Fail closed.** If the channel cannot be read, `check` and `claim` return
+  `503` rather than reporting the node free: an unreadable channel looks exactly
+  like "nobody has claimed anything". An agent's registry JWT is presented on
+  that read too, so a bus that gates reads does not look like a dead channel.
+- **Claim is both halves or neither.** The cluster lease is rolled back if the
+  bus post fails, so a peer that only watches the bus never disagrees with the
+  local scheduler about who holds the node. Release is ordered the same way: the
+  line is posted BEFORE the local lease is freed, so a failed post leaves the
+  lease intact rather than freeing a node peers still see as claimed.
+- **An operator's release is attributed to the holder whose claim it closes.**
+  A lease taken through `/claim` can also be freed by an explicit `lease_id` —
+  by its holder, or by an operator (`_may_act_on`). The node-scoped form (no
+  `lease_id`) only ever selects the caller's OWN lease: a body `holder` is
+  display text and never an identity, so it cannot be used to select someone
+  else's lease. Since a bus claim is keyed
+  on its **author**, the operator's `[GPU RELEASE]` is posted as the freed
+  holder, not as the operator (an admin session may set an explicit `from`, see
+  *Posting to the coordination bus*); the response reports both, `holder` (who
+  acted) and `released_holder` (whose claim the line closes). A bus that
+  authenticates senders refuses the substitution, and because the line is still
+  posted before the local lease is freed, the override then fails loudly
+  (`502`) with the lease intact rather than leaving the two views disagreeing.
+- **Keep-alive is the TTL, not a promise.** A cluster-worker lease expires after
+  `ttl_seconds` (default 300, capped at 3600) unless renewed via `/renew`; a
+  crashed or idle holder therefore frees the node without anyone releasing it.
+  An unbounded TTL would let one agent take the shared GPU permanently, so the
+  cap is enforced by the request model.
+- **A claim carries its own expiry, and the fold honours it.** `/claim` publishes
+  `expires=<unix ts>` on the line — the backing lease's expiry when the node is a
+  cluster worker, else the TTL it was asked for — rounded up, so a published
+  expiry can never precede the reservation it describes. `/renew` reposts the
+  claim as it extends the lease, on the channel the claim was made on (the
+  channel is an input to the *claim*, never to its renewal); if that repost
+  fails, the local extension is rolled back, so peers and this controller still
+  agree and the holder can retry. A fold drops a claim whose published expiry has
+  passed, exactly as the cluster lease's TTL frees its reservation, so a holder
+  that crashed or stopped keeping alive no longer blocks the card until its
+  claim ages out of the fold window. Keep-alive therefore means re-POST `/claim`
+  or `/renew` while you hold the card; a claim posted by hand without an
+  `expires=` has no TIME-based expiry — it is closed by a RELEASE — but it is
+  still subject to the fold window below, so a long-lived hand-posted claim has
+  to be reposted periodically like any other. That is what the interim protocol
+  in #893 relies on.
+- **A claim is only visible inside the channel fold window** (the newest 500
+  messages). For a load that outlives the chatter around it, re-POST `/claim`
+  periodically: it is idempotent (it extends the lease and reposts the line,
+  which the fold treats as a replacement, never a second claim).
+- **`node` labels** resolve to a cluster worker by name, by its URL host, or to
+  the local controller for `local`/`localhost`/this hostname. A node this
+  controller does not know is bus-governed only (no local lease), and its CHECK
+  is reported as `vram_verified: false` rather than as free.
+- The channel defaults to `gpu`; point every agent at the same thread with
+  `TAOS_A2A_GPU_CHANNEL`.
+
+`check` returns `admitted`, `blockers`, `free_mb`, `capacity_mb`, `claimed_mb`,
+`vram_verified`, `reason`, and the `claims` it folded. `claim` returns the
+`lease_id` (when the node is a cluster worker) and the exact `line` posted.
+
 ## Bus restarts during a controller update
 
 `POST /api/settings/update` on a host that also runs taOSmd locally (config
@@ -1453,6 +1559,14 @@ worker lane is therefore refused on `POST` (it lacks the create grant, `403`)
 and authorised on `GET`. `tests/test_routes_task_checklist.py` pins this scope
 split directly, not behind an xfail.
 
+Shared-GPU leases (taOS #893). `GET /api/a2a/gpu/check` (scope `a2a_receive`)
+and `POST /api/a2a/gpu/{claim,release,request,renew}` (scope `a2a_send`). The
+route resolves the acting identity from the token and forces the bus `from` to
+the identity the token proves, so an agent can only claim/release GPU capacity
+for itself. See *Shared-GPU leases (`/api/a2a/gpu/*`)* above for the protocol,
+the admission rules, and why CHECK/CLAIM fail closed when the channel is
+unreadable.
+
 Container provisioning request (P1 + P2, agent-container-provisioning spec):
 
 - `POST /api/containers/requests` and `POST /api/container-requests` -- an active agent submits a container provisioning request with its own registry JWT. The route resolves the canonical_id from the token (never from the request body) and applies the provisioning policy (per-agent quota + threshold). Under quota the request is auto-approved; over quota it lands in `pending-approval`; over threshold it is escalated to a Decisions-app item for Jay. This is an identity-only check (no scope grant required), matching the scope-request create flow's use of `check_agent_identity`.

tests/test_cluster.py

@@ -422,6 +422,49 @@ async def test_draining_workers_excluded_from_lease_claim(self):
         )
         assert lease is None
 
+    async def test_restore_lease_expiry_refuses_to_clobber_a_newer_renewal(self):
+        """A rollback must not undo a renewal that landed in the meantime.
+
+        `gpu_renew` posts to the bus outside `_lease_lock`, so another renewal
+        can extend the lease while the first one's post is in flight. The
+        rollback is therefore a compare-and-set on the expiry THIS caller
+        attempted (Kilo/CR review of #2988).
+        """
+        mgr = ClusterManager()
+        await mgr.register_worker(_make_worker("gpu-box", url="http://gpu-box:9000"))
+        mgr.get_worker("gpu-box").free_vram_mb = 8000
+        lease = await mgr.claim_lease(
+            resource_id="gpu-box:gpu-cuda-0", caller="a2a:@peer", ttl_seconds=300
+        )
+        assert lease is not None
+        previous = lease.expires_at
+
+        ours = await mgr.renew_lease(lease.lease_id, ttl_seconds=600)
+        attempted = ours.expires_at
+        # Another renewal moves the expiry on while our bus post is in flight.
+        later = await mgr.renew_lease(lease.lease_id, ttl_seconds=900)
+        assert later.expires_at > attempted
+
+        restored = await mgr.restore_lease_expiry(
+            lease.lease_id, previous, attempted_expiry=attempted
+        )
+        assert restored is False
+        assert mgr.get_leases()[0].expires_at == later.expires_at
+
+        # With nothing superseding us, the rollback applies.
+        restored = await mgr.restore_lease_expiry(
+            lease.lease_id, previous, attempted_expiry=later.expires_at
+        )
+        assert restored is True
+        assert mgr.get_leases()[0].expires_at == previous
+
+        # A lease that is gone needs no restore.
+        await mgr.release_lease(lease.lease_id)
+        restored = await mgr.restore_lease_expiry(
+            lease.lease_id, previous, attempted_expiry=previous
+        )
+        assert restored is False
+
     # ── Worker-initiated drain (taOS #890 C2) ──────────────────────────
 
     async def test_heartbeat_status_draining_triggers_worker_self_drain(self):