reconcile-scheduler has never completed: publish deadline silently drops the alphabetical tail of sites

[raisedadead]

Summary

reconcile-scheduler has failed on every run since the bounded publish deadline landed in v1.4.0 (091887f, 2026-07-14). Production Hatchet run history: 22 runs, 22 FAILED, zero COMPLETED.

Cause

The workflow budgets reconcilePublishFloor + n*reconcilePublishPerSite — 4s + 150ms/site — for the entire publish loop (cmd/artemis/gcworkflows.go:75-80). Real Hatchet publish latency is several times that, so the loop exhausts its own deadline and breaks partway through the site list (cmd/artemis/gcworkflows.go:121-127), returning firstErr and failing the run.

Because Snapshot().Sites() returns a sort.Strings-sorted slice (internal/registry/valkey/reader.go:80), the truncation point is deterministic — the same alphabetical tail is skipped every single day.

Impact

Measured against production run history (v1_statuses_olap / v1_payloads_olap):

8 of 64 registered sites have never been reconciled since 2026-07-15: test, the-story-of-the-moon, titrations, tower-of-hanoi, typing-coach, verb-roulette, verb-snap, wave-forge

Coverage is also degrading over time:

day	sites published	stopped at
2026-08-01	53	sudoku
2026-08-03	40	projectile-motion
2026-08-04	32	omotenashi-training
2026-08-05	33	palette-contrast-checker

Drift on skipped sites — orphaned R2 objects, aliased-missing deploys — is never detected or healed.

Fix direction

• Recalibrate the per-site budget against measured publish latency, and/or publish concurrently.
• Prefer one generous ceiling over a per-item budget that truncates silently.
• When the loop breaks early, log the skipped count and sites. Today the failure reports only context deadline exceeded — nothing says N sites were dropped.
• Consider rotating iteration order so truncation cannot pin the same tail indefinitely.

Sentry: ARTEMIS-A.