Claim hot-path slows ~33% with deadline rescue enabled at high concurrency

[hardbyte]

Context

At 1×256 worker concurrency, depth-target driving awa.queue_lanes near saturation, enabling per-claim deadline rescue costs ~33% steady-state throughput and ~50% p99 latency on the claim hot path. The cost is distributed across the whole claim path rather than concentrated in a single rescue-specific query, which suggests a structural rather than additive overhead.

Reproduction shape: awa-bench long_horizon scenario, single replica, 256 workers, depth-target = 4000, JOB_WORK_MS=1, producer-mode=depth-target, 30 s warmup + 120 s clean. Two cells, identical except LEASE_DEADLINE_MS:

	rescue OFF (LEASE_DEADLINE_MS=0)	rescue ON (library default)	Δ
completion_rate	5,419 jobs/s	3,649 jobs/s	−33 %
offered (depth-target burst)	3,905 jobs/s	2,752 jobs/s	−30 %
end_to_end_p99_ms	100 ms	151 ms	+51 %

What pg_stat_statements says

pg_stat_statements_reset() immediately before each cell, snapshot at +90 s into the clean phase. Top hot-path queries (mean_exec_time, ms):

query	OFF mean	ON mean	Δ
UPDATE awa.queue_enqueue_heads SET next_seq = next_seq + $3 ...	14.8	18.0	+22 %
INSERT INTO awa.queue_enqueue_heads ON CONFLICT DO NOTHING	5.6	7.3	+30 %
SELECT ... FROM awa.claim_ready_runtime(...)	1.28	1.60	+25 %
INSERT INTO awa.queue_lanes ON CONFLICT DO NOTHING	2.9	3.3	+14 %
INSERT INTO awa.queue_claim_heads ON CONFLICT DO NOTHING	2.8	3.3	+18 %
UPDATE awa.queue_lanes ... deltas ...	2.2	2.5	+13 %
UPDATE awa.queue_lanes SET available_count = ...	1.7	2.0	+18 %

Two things this doesn't show:

1. No new query in the top-50 with rescue ON. If the rescue scanner were a periodic standalone scan firing visibly, it should appear. It doesn't — either it's inside the regular claim path (e.g. force-close happens inline in claim_ready_runtime), or it's batched at a low enough cadence that even with 33 % throughput cost it stays below the noise floor.
2. The cost is uniform, not localized. Every hot query is 13–30 % slower with rescue ON. That's the fingerprint of contention on a shared resource (lock contention, buffer-cache pressure, index walk cost), not "the rescue path itself adds N ms per claim."

The body of claim_ready_runtime documents the mechanism in a comment:

deadline_at is the per-claim deadline when the queue has a non-zero deadline_duration; the deadline-rescue path scans expired rows (anti-joined with closures and leases — same disambiguation as the heartbeat-rescue path) and force-closes them.

Hypothesis

The most consistent explanation for uniform per-query slowdown without a visible new query is:

awa.lease_claims working set inflates when rescue is on, and every other query that touches lease_claims (every claim, every completion) pays the cost in index lookup time / page fetches / lock contention.

That'd happen if claims awaiting force-close linger in lease_claims longer than they would when deadline_at IS NULL (where the table only ever sees a row for the duration of an in-flight job).

Equally consistent secondary hypothesis: the rescue scanner is doing a sequential scan or partial-index miss across lease_claims, holding short locks that serialize against the regular claim INSERT/UPDATE path.

Proposed experiments

In rough order of cheap → less cheap:

1. pg_stat_user_tables.n_live_tup snapshot of awa.lease_claims at end-of-clean in both cells. If rescue ON shows materially more rows than rescue OFF at steady state, the working-set inflation hypothesis is confirmed. ~5 minutes of work.
2. Add awa.lease_claims (deadline_at) WHERE deadline_at IS NOT NULL partial index and re-run rescue ON. If the per-claim latency tax drops, the rescue scanner is doing an avoidable seqscan / full-index walk.
3. Tune the rescue scanner's batch size / wake interval (or, if it's currently inline in claim_ready_runtime, hoist it out to a background task with its own cadence). Look for the knee where the scanner reaps fast enough to keep lease_claims small but doesn't dominate the claim path.
4. Flame graph of awa-bench at 1×256 with rescue on vs off. Would directly confirm whether the time goes to PG roundtrips, lock waits, or in-process work in the awa-worker claim loop.

The first two are non-invasive and should be enough to localize the cost. (3) and (4) are needed if the first two come back ambiguous.

Safety considerations

• Don't disable deadline_at writes by default to fix this. Per-claim deadline rescue is the documented fallback path for stuck workers under correctness invariants the chaos suite relies on; the wrong fix would be making the bench look better at the cost of a real reliability mechanism.
• The fix should not change the visible behavior of force-close. If a job's deadline expires, it must still be force-closed and re-eligible for claim. Both inline (current) and background (proposed) implementations need to preserve this.
• The partial-index proposal needs a migration plan. awa.lease_claims is partitioned in current schema; the partial index needs to be defined per-partition and on the master.
• If the working-set hypothesis is right, there's an upstream knock-on for chaos scenarios. A larger steady-state lease_claims table also means slower force-close on a true wedge, because the rescue scan walks more rows. Worth measuring how long it takes a wedged claim to be force-closed at high concurrency before merging any tuning that lets the table grow.

Reproducer

awa-bench adapter from the postgresql-job-queue-benchmarking repo at branch bench/2026-05-07-awa-alpha6-pgque-rc1:

docker compose up -d --wait
LEASE_DEADLINE_MS=0 \
  uv run bench run --systems awa --replicas 1 \
    --producer-rate 50000 --producer-mode depth-target --target-depth 4000 \
    --worker-count 256 \
    --phase warmup=warmup:30s --phase clean=clean:120s

# repeat without LEASE_DEADLINE_MS to use library default

Tested against awa 0.6.0-alpha.6 and 0.6.0-alpha.7; both reproduce the same shape.

pg_stat_statements snapshots are committed under results/2026-05-08-rescue-perf-probe/snapshots/ for direct inspection.

[hardbyte]

Wait-event evidence — 2026-05-12

Built a queue-storage-specific contention probe (test_queue_storage_enqueue_contention in awa/tests/benchmark_test.rs) plus a server-side pg_stat_activity sampler that calls pg_stat_clear_snapshot() between iterations (the default stats_fetch_consistency='cache' was hiding the live state otherwise). Ran the sampler at 25 ms intervals during a 16-producer same-queue queue-storage enqueue load on the local machine.

Wait-event breakdown of producer time (1,683 samples across 17 backends, sampler excluded):

Wait type	Wait event	% producer time
Lock	transactionid	63.5%
Lock	tuple	29.6%
(CPU)	running	4.4%
IO	WalSync	1.1%
Client	ClientRead	1.1%
IO	DataFileExtend	0.3%

Of the Lock-class waits, 93.4% are attributed to the enqueue_head_update query (the UPDATE queue_enqueue_heads SET next_seq = next_seq + $3 ... RETURNING next_seq - $3).

This refutes the earlier hypothesis that the awa.lease_claims working-set inflation was the dominant cost on the rescue-ON cell. The cost is row-lock contention on the single hot queue_enqueue_heads row, not lease-claims pressure. The deadline-rescue overhead measured in this issue may still have a secondary effect on lease_claims, but the headline 33% slowdown reproducibility-wise comes from contention on the head row.

Implications for #246

1. The proposed lease_claims (deadline_at) WHERE deadline_at IS NOT NULL partial index is still worth a measurement — it could help on the rescue-ON path independently — but it isn't the highest-value lever.
2. Sharding queue_enqueue_heads into N rows per (queue, priority) is the highest-value lever, because it directly targets where 93% of producer wall-clock time goes. This work is now in progress on a feature branch and will land as migration v017.

Local A/B on the same probe (16 producers × 15 k jobs × 3 runs, queue-storage same-queue):

Variant	Mean throughput
Baseline (post-v016, no in-process lane cache)	29,793 jobs/s
With in-process ensure_lane cache (just merged on a branch)	48,404 jobs/s (+62%)

The cache eliminates three INSERT … ON CONFLICT DO NOTHING round-trips on subsequent enqueues to a known lane. With the cache in place, the next bottleneck is unambiguously the head-row UPDATE — which is what Phase D will address.

Once v017 lands, I'll re-run the same probe and update this issue with the new wait-event distribution; the expectation is that Lock:transactionid drops to single-digit percent and the next-largest wait is IO:WalSync (currently 1.1%).

[hardbyte]

Phase D landed — sharded queue_enqueue_heads

Migration v017 adds an enqueue_shard column to queue_enqueue_heads, queue_claim_heads, and ready_entries, keyed by a per-queue awa.queue_meta.enqueue_shards tunable (default 1, range 1..=64). Producers rotate across the shard rows so row-lock contention on a single hot row is spread across N independent rows. S=1 is observationally identical to the pre-v017 layout — the migration is a structural no-op for behaviour at default.

Local A/B sweep — 16 producers × 15 k jobs/producer, same queue (in-tree test_queue_storage_enqueue_contention, 3 runs per cell)

enqueue_shards	Mean throughput (jobs/s)	vs S=1
1 (default)	41,377	1.00×
2	72,704	1.76×
4	126,374	3.05×
8	200,359	4.84×

Combined with the in-process ensure_lane cache that landed in the same release, the total improvement on the queue-storage same-queue enqueue path is ~6.7× (30 k jobs/s pre-v016+cache → 200 k jobs/s at S=8). Scaling stays near-linear up to S=8 on this hardware (2 producers per shard at S=8); the knee where WAL bandwidth becomes the new bottleneck is past S=8 with this concurrency.

Implications for deadline-rescue overhead (the original #246 question)

The published 33% slowdown was attributed in this issue to working-set inflation on awa.lease_claims. After Phase D the producer wall-clock per batch is dominated by WAL flush, not row-lock waits. Whether the rescue overhead persists at the new throughput floor (Lock:transactionid no longer 93% of time) needs a fresh measurement — I'd expect the absolute overhead to be similar (the rescue scan walks the same number of rows), but the relative impact on throughput to be much smaller.

The partial index proposal from this issue (awa.lease_claims (deadline_at) WHERE deadline_at IS NOT NULL) is still cheap to land independently if a fresh probe shows the scan cost; deferring until the post-Phase-D benchmarks tell us where time goes.

Next step

Re-run the bench-repo sweep against main with v017 applied. With enqueue_shards = 1 (default) we should see the same per-row contention pattern, plus the +62% from the in-process cache. With UPDATE queue_meta SET enqueue_shards = 8 WHERE queue = 'awa-bench' we should see the 4–5× lift demonstrated locally.

[hardbyte]

Bench-repo Awa-only evidence against current main — 2026-05-17

Ran the companion hardbyte/postgresql-job-queue-benchmarking harness against
Awa main after #261.

Setup:

• Awa commit: 1e422488f961 (main, includes Fix queue-storage shard joins and nightly chaos races #261)
• Benchmark checkout: 93677cab92da with the current local harness changes for
  adapter env passthrough, wait-event sampling, WAL/job, commits/job, and
  Awa-specific setup in the adapter layer
• Postgres image: postgres:18.3-alpine
• awa-bench was freshly release-built against the local Awa checkout before
  the sweep; subsequent cells used --skip-build
• Wait-event sampling enabled
• Shape: 2 replicas, 256 workers/replica, producer target 5000 jobs/s per
  replica (10000 jobs/s aggregate target), PRODUCER_BATCH_MAX=512,
  10s warmup + 45s clean phase, sample cadence 2s

Cell	Actual enqueue/s	Complete/s	E2E p99	Median depth	Peak dead tuples	WAL/job	Commits/job	Top wait
enqueue_shards=1 default	7,743	7,506	3,047 ms	21,945	400	1,962 B	0.090	LWLock:WALWrite
enqueue_shards=4	7,965	8,020	196 ms	125	238	2,040 B	0.114	LWLock:WALWrite
enqueue_shards=8	7,902	7,899	208 ms	126	225	2,008 B	0.112	LWLock:WALWrite
enqueue_shards=8, completion batch 1024, flush 20ms	8,154	8,213	224 ms	126	218	1,893 B	0.099	LWLock:WALWrite

Result dirs:

• results/custom-20260517T092459Z-dafd8b — enqueue_shards=1
• results/custom-20260517T092628Z-6cbbbc — enqueue_shards=4
• results/custom-20260517T092757Z-c68713 — enqueue_shards=8
• results/custom-20260517T092925Z-2198da — enqueue_shards=8 plus completion batching

Interpretation:

• The default single enqueue-head shard does not hold the 10k/s aggregate target
  steady on this machine: actual enqueue is ~7.7k/s, completion is ~7.5k/s, the
  queue builds to ~22k jobs, and p99 rises to ~3s.
• enqueue_shards=4 removes the visible hot-head backlog at the same offered
  shape: completion rises to ~8.0k/s, median depth falls to ~125, and p99 falls
  to ~200ms.
• enqueue_shards=8 is not materially better than 4 in the full end-to-end
  path, which says the bottleneck has moved off the single enqueue-head row.
• Completion batching is the best result in this sweep: ~8.2k/s completed with
  lower WAL/job and lower commits/job. It still does not make this run absorb
  the full 10k/s target, so the remaining work is worker-side WAL/transaction
  budget and terminal-history pressure, not more enqueue-head sharding.
• The top wait in all cells is now LWLock:WALWrite, not the earlier
  Lock:transactionid / tuple-lock signature. That matches the expected
  post-v017 story: sharding addressed the hot-head lock contention, and the next
  limiting resource is durable write/commit pressure.

This also reframes the original deadline-rescue question. These runs use the
default rescue behavior; the e2e bottleneck at current main is no longer
primarily explained by the earlier rescue-on/off delta. A targeted rescue
on/off probe may still be useful, but it is now a second-order measurement
behind enqueue sharding and completion/WAL pressure.