fix(service-queue): reclaim a running job whose lease ran out, and renew a live handler's lease

.changeset/30228-queue-reclaim-expired-lease.md

@@ -0,0 +1,15 @@
+---
+'@objectstack/service-queue': patch
+---
+
+fix(service-queue): a job whose worker died mid-handler is reclaimed once its lease runs out, instead of staying `running` forever
+
+Clause-②: no — restores the documented `leaseMs` reclaim; no accept set or export widens
+Metadata-protocol: none
+
+`DbQueueAdapterOptions.leaseMs` documented a lease "before another worker may reclaim", but the claim read only `pending` rows, so a worker that crashed, was OOM-killed or was redeployed mid-handler left its `sys_job_queue` row `running` for good: never retried, never dead-lettered.
+
+- **Reclaim.** A poll tick reads the table's `running` rows (one read for every queue, at most once per lease window) and returns each whose `locked_until` has passed to the retry path. The lost run counts as an attempt: the row goes back to `pending` on its backoff, or to `dlq` once `maxAttempts` is spent, with `last_error` saying the lease expired. The write is a compare-and-set on the expired lease, so a holder that renewed in the meantime keeps its run.
+- **Renewal.** While a handler runs, its worker moves its lease a full window ahead every third of `leaseMs`, so a slow but live handler is never reclaimed and run twice. A handler must not block the event loop for most of a lease window.
+- **Sole writer.** With `soleWriter` no other process claims rows, so the reclaim reads once per boot (finding what a crashed previous process left behind) and again only when one of this process's own claims was left unsettled by a failed write and its lease runs out. `nextDueAt` folds in those leases and the earliest live lease the last read saw, at no read of its own, so the loop wakes for them instead of at the idle ceiling.
+- **Cost.** Without `soleWriter`, an idle queue pays one more indexed read per lease window (default 30 s): an idle hour at the defaults goes from 125 to 246 reads. With `soleWriter`, one more read per boot: a fresh kernel's first minute goes from 2 to 3 reads, and an idle hour carries no other.

content/docs/permissions/tenant-audit-census.mdx

@@ -132,7 +132,7 @@ are reported as `undecidable` rather than assumed either way.
 
 The same holds twice over for the context. An options argument spelled as a
 literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
-forwarding shim cannot, and **55 of the 259 sites are spelled that way**. A
+forwarding shim cannot, and **55 of the 261 sites are spelled that way**. A
 context resolved from an inline literal or a local `const` can be tested for
 `isSystem`; one arriving from a helper call cannot.
 
@@ -197,10 +197,10 @@ reproduce them. Where it disagrees, it disagrees on the page:
 
 | carried figure | where it survives | this census |
 | :--- | :--- | ---: |
-| 175 write call sites | quoted in the merged changeset | **259** |
+| 175 write call sites | quoted in the merged changeset | **261** |
 | 24 carrying no tenant context | quoted in the merged changeset | **2** provable and tenancy-enabled; **29** more whose options argument is unreadable |
-| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **182 of 259** decidable, **77** undecidable |
-| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 144 decidably elevated, 0 decidably not, 107 undecidable |
+| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **184 of 261** decidable, **77** undecidable |
+| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 144 decidably elevated, 0 decidably not, 109 undecidable |
 | 141 and 132, two independent re-derivations | the card that filed this work | — |
 
 **The differences are not reconciled, and deliberately so.** The old census's
@@ -211,17 +211,17 @@ at any commit.
 
 Two structural facts do plausibly widen this reading against any hand or regex
 one, and both are counted in the generated tables below: the 62 sites reached
-through an erased (`any`) receiver, and the 66 that name their object through a
+through an erased (`any`) receiver, and the 68 that name their object through a
 `const` rather than inline. An instrument that read either the way a person does
 would report a smaller number and would not say so.
 
 The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
 figure has no surviving corroboration anywhere in the tree.** This census reads
-144 of 259 (56%) as decidably elevated, with 107 more whose elevation is a
+144 of 261 (55%) as decidably elevated, with 109 more whose elevation is a
 run-time fact — so the claim is neither confirmed nor refuted, and the honest
 answer is that a static reading cannot settle it.
 
-⇒ **Cite `2 / 259`, and say what it is**: the sites whose options argument was
+⇒ **Cite `2 / 261`, and say what it is**: the sites whose options argument was
 READ and holds no tenant context, against a decidably tenancy-enabled object.
 That is the control's provable yield surface. ⛔ Do not cite it as "the sites
 without tenant context" — **29 further sites** have an options argument this
@@ -233,29 +233,29 @@ cannot read, and they are neither in nor out.
 
 | what | count |
 | :--- | ---: |
-| write call sites on the application surface | **259** |
-| …whose object name is statically decidable | 182 |
+| write call sites on the application surface | **261** |
+| …whose object name is statically decidable | 184 |
 | …whose object name is chosen at run time | 77 |
-| …against an object with tenancy ENABLED | 176 |
+| …against an object with tenancy ENABLED | 178 |
 | …against an object that declares tenancy off | 6 |
-| threading a tenant context | 196 |
+| threading a tenant context | 198 |
 | PROVABLY carrying none (options read, no context key) | **8** |
 | …of those, against a decidably tenancy-enabled object | **2** |
 | options argument UNREADABLE — may or may not carry one | 55 |
 | …of those, against a decidably tenancy-enabled object | 29 |
 | threading a decidably ELEVATED (`isSystem`) context | 144 |
 | threading a context that is decidably NOT elevated | 0 |
-| threading a context whose elevation is a run-time fact | 107 |
+| threading a context whose elevation is a run-time fact | 109 |
 
 | how the instrument reached the site | count |
 | :--- | ---: |
-| receiver carried a readable engine type | 197 |
+| receiver carried a readable engine type | 199 |
 | receiver erased, placed by the object NAME | 52 |
 | receiver erased, placed by an `object: string` PARAMETER | 5 |
 | receiver erased, placed by an `UNTYPED_RECEIVERS` row | 5 |
 
 | object name spelled inline | 116 |
-| object name spelled through a `const` | 66 |
+| object name spelled through a `const` | 68 |
 | object name is an `object: string` parameter | 7 |
 | object name is some other run-time expression | 70 |
 
@@ -307,13 +307,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
 their values are not compared. The reasoning, and the measurement behind it,
 are in `scripts/check-tenant-audit-census.mjs`.
 
-Measured on 2026-11-03 at `1c86b45c5`.
+Measured on 2026-11-05 at `e40e04854`.
 
 | corpus scale (not enforced) | count |
 | :--- | ---: |
-| tracked non-test sources scanned | 856 |
-| engine-shaped types recognised | 103 |
-| declared objects in the registry | 126 |
-| same-named calls subtracted as non-engine | 226 |
+| tracked non-test sources scanned | 871 |
+| engine-shaped types recognised | 105 |
+| declared objects in the registry | 127 |
+| same-named calls subtracted as non-engine | 229 |
 
 {/* END GENERATED: tenant-audit-census */}

docs/audits/2026-08-tenant-audit-write-call-sites.counts.md

@@ -33,19 +33,19 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution.
 
 | Measure | Value |
 |---|---:|
-| Write call sites | 259 |
-| Object name statically decidable | 182 |
+| Write call sites | 261 |
+| Object name statically decidable | 184 |
 | Object name chosen at run time | 77 |
-| Against a tenancy-enabled object | 176 |
+| Against a tenancy-enabled object | 178 |
 | Against an object declaring tenancy off | 6 |
-| Threading a tenant context | 196 |
+| Threading a tenant context | 198 |
 | Provably carrying none | 8 |
 | …and decidably tenancy-enabled | 2 |
 | Options argument unreadable | 55 |
 | …and decidably tenancy-enabled | 29 |
 | Threading a decidably elevated context | 144 |
 | Threading a decidably non-elevated context | 0 |
-| Threading a context of undecidable elevation | 107 |
+| Threading a context of undecidable elevation | 109 |
 
 ## Subtractions the census could NOT defend — enforced
 
@@ -90,14 +90,14 @@ holds still. They are required to be HERE and to say WHEN they were true;
 their values are not compared. The reasoning, and the measurement behind it,
 are in `scripts/check-tenant-audit-census.mjs`.
 
-Measured on 2026-11-03 at `1c86b45c5`.
+Measured on 2026-11-05 at `e40e04854`.
 
 | corpus scale (not enforced) | count |
 | :--- | ---: |
-| tracked non-test sources scanned | 856 |
-| engine-shaped types recognised | 103 |
-| declared objects in the registry | 126 |
-| same-named calls subtracted as non-engine | 226 |
+| tracked non-test sources scanned | 871 |
+| engine-shaped types recognised | 105 |
+| declared objects in the registry | 127 |
+| same-named calls subtracted as non-engine | 229 |
 
 ## Every site
 
@@ -248,7 +248,7 @@ Measured on 2026-11-03 at `1c86b45c5`.
 | `packages/services/service-messaging/src/sql-outbox.ts` | `update` | `this.objectName` | undecidable | options unreadable | 1 |
 | `packages/services/service-queue/src/db-queue-adapter.ts` | `delete` | `sys_job_queue` | enabled | context, elevation undecidable | 2 |
 | `packages/services/service-queue/src/db-queue-adapter.ts` | `insert` | `sys_job_queue` | enabled | context, elevation undecidable | 1 |
-| `packages/services/service-queue/src/db-queue-adapter.ts` | `update` | `sys_job_queue` | enabled | context, elevation undecidable | 6 |
+| `packages/services/service-queue/src/db-queue-adapter.ts` | `update` | `sys_job_queue` | enabled | context, elevation undecidable | 8 |
 | `packages/services/service-settings/src/config-change-audit.ts` | `insert` | `sys_audit_log` | enabled | context, elevation undecidable | 1 |
 | `packages/services/service-settings/src/settings-service-plugin.ts` | `insert` | `objectName` | undecidable | options unreadable | 1 |
 | `packages/services/service-settings/src/settings-service-plugin.ts` | `update` | `objectName` | undecidable | options unreadable | 2 |