fix(upstream-watch): reconcile main pacing with develop safeguards

.github/workflows/upstream-watch.yml

@@ -58,8 +58,10 @@ jobs:
           set -e
           {
             echo "## Upstream watch preview (exit $code)"
-            jq -r 'if .blind then "blind \(.blind): \(.error // "unknown error"), could not check \(.fetchErrors | length)" else "since \(.since) (\(.sinceSource)), new records \(.records | length), could not check \(.fetchErrors | length), blind \(.blind), notice \(.notice.post), would fire \(.wouldFire | length)" end' watch.json
+            jq -r 'if .blind then "blind \(.blind): \(.error // "unknown error"), could not check \(.fetchErrors | length), dispatch errors \(.dispatchErrors | length), deferred \(.deferred | length)" else "since \(.since) (\(.sinceSource)), new records \(.records | length), could not check \(.fetchErrors | length), dispatch errors \(.dispatchErrors | length), blind \(.blind), notice \(.notice.post), would fire \(.wouldFire | length), deferred \(.deferred | length)" end' watch.json
             jq -r '(.wouldFire // [])[] | "- would fire \(.id) \(.version) \(.branch)"' watch.json
+            jq -r '(.deferred // [])[] | "- deferred \(.id) \(.version) \(.branch)"' watch.json
+            jq -r '(.fired // [])[] | "- observed session before ledger failure: \(.id) \(.fields.session)"' watch.json
             echo; echo '```text'; cat errors.txt; echo '```'
           } >> "$GITHUB_STEP_SUMMARY"
           exit "$code"
@@ -102,8 +104,10 @@ jobs:
           set -e
           {
             echo "## Upstream watch (exit $code)"
-            jq -r 'if .blind then "blind \(.blind): \(.error // "unknown error"), could not check \(.fetchErrors | length), dispatch errors \(.dispatchErrors | length)" else "since \(.since) (\(.sinceSource)), new records \(.records | length), could not check \(.fetchErrors | length), dispatch errors \(.dispatchErrors | length), blind \(.blind), commit \(.commit // "none"), would fire \(.wouldFire | length)" end' watch.json
+            jq -r 'if .blind then "blind \(.blind): \(.error // "unknown error"), could not check \(.fetchErrors | length), dispatch errors \(.dispatchErrors | length), deferred \(.deferred | length)" else "since \(.since) (\(.sinceSource)), new records \(.records | length), could not check \(.fetchErrors | length), dispatch errors \(.dispatchErrors | length), blind \(.blind), commit \(.commit // "none"), would fire \(.wouldFire | length), deferred \(.deferred | length)" end' watch.json
             jq -r '(.wouldFire // [])[] | "- would fire \(.id) \(.version) \(.branch)"' watch.json
+            jq -r '(.deferred // [])[] | "- deferred \(.id) \(.version) \(.branch)"' watch.json
+            jq -r '(.fired // [])[] | "- observed session before ledger failure: \(.id) \(.fields.session)"' watch.json
             echo; echo '```text'; cat errors.txt; echo '```'
             jq -r '.notice.body' watch.json
           } >> "$GITHUB_STEP_SUMMARY"

docs/archive/2026-09-29-plan-main-watch-reconciliation.md

@@ -0,0 +1,47 @@
+# Main watcher reconciliation
+
+## Status
+
+Implemented and independently reviewed through `7c2f3054`. The controller integrated
+validated V6 develop at `3bbee599`; the reviewed watcher files were unchanged by
+that merge. All eight local gates passed on that exact clean commit: 6,440 unit
+tests passed, seven skipped, legacy suites passed, and browser checks passed.
+Coverage was 94.18% lines, 83.80% branches and 93.50% functions. Feature PR CI,
+squash integration and final human main review remain separate gates at archival.
+No real routine trigger was performed. The limits below describe the worker scope.
+
+## Scope and acceptance
+
+Combine develop PR observation and blind reporting with main #280 dispatch pacing.
+Preserve pending eligibility, the strict seven day observation window, two recorded
+firings per thread, three day cooldown, three fixes per run, and 15 second spacing.
+Then restrict trigger retries to documented HTTP 500/503, sanitize token-bearing
+error metadata, preserve deferred work in bounded previews and blind results, and
+align the living guide and workflow summaries. Use focused synthetic regression
+tests before behavior changes and scoped static checks.
+
+## Ownership and policy receipt
+
+The controller assigned sole writing ownership of watcher source, workflow, tests,
+and guide in `fix/main-watch-reconciliation`, and authorized the prepared two-parent
+merge commit followed by three conventional unit commits. Shared manifests,
+lockfiles, V6 fixtures, develop/main integration, archive index changes, publication,
+and final full gates remain controller-owned. This plan is ready for the controller
+to archive with its index update in the completing pull request.
+
+## Worker limits
+
+No real trigger, provider turn, external mutation, full suite, UI tests, pnpm,
+installed CLI changes, user data writes, or delegated workers. Inject fetch,
+execution, and sleeps. Retry evidence does not establish exactly-once execution or
+absence of server-side sessions. Deferred work remains eligible for later checks;
+repeated earlier failures can starve it.
+
+## Units
+
+1. Reconcile and commit both parent contracts; establish focused baseline.
+2. Test and fix bounded retry and sanitized error diagnostics.
+3. Test and fix preview cap, deferred blind result, and later-run discoverability.
+4. Clarify guide, workflow preview/record summaries, and evidence limits.
+
+Record command evidence and limitations in the ignored worker implementation report.

docs/archive/README.md

@@ -69,6 +69,7 @@ reconfirmed by this metadata audit. The per-file inventory and limitations are r
 | File | Original location | What it was | Why it's historical |
 |---|---|---|---|
 | [2026-09-28-plan-follow-ups-v2.md](2026-09-28-plan-follow-ups-v2.md) | `docs/plans/2026-09-28-follow-ups-v2.md` | V4 product, CLI, memory, process-lifecycle and upstream integration follow-ups. | All eight local gates and independent whole-branch review passed at `29152654`; final-head PR CI and squash integration were pending at archival. Conditional Ruflo #3419 guidance remains deferred; native Windows AQE was not tested. |
+| [2026-09-29-plan-main-watch-reconciliation.md](2026-09-29-plan-main-watch-reconciliation.md) | `docs/plans/2026-09-29-main-watch-reconciliation.md` | Completed main #280 and develop watcher reconciliation | Both contracts preserved; bounded documented retry, sanitized diagnostics and faithful deferred preview. Independent review and all eight local gates passed through `3bbee599`; PR CI and squash integration remained pending at archival. Current contract: [Upstream watch](../upstream-watch.md). |
 | [2026-09-28-plan-usage-accuracy.md](2026-09-28-plan-usage-accuracy.md) | `docs/plans/2026-09-28-usage-accuracy.md` | Completed V6 usage and session evidence plan | All 24 units independently accepted; whole-branch review and all eight local gates passed through `1c02db91`. Feature PR CI and develop integration remain separate gates at archival. Current contracts: [Usage metrics](../usage-scorecard-metrics.md) and [ADR-0060](../adr/0060-session-surface-initiator-and-product-names.md). |
 | [2026-09-29-plan-v6-surfaces-ui.md](2026-09-29-plan-v6-surfaces-ui.md) | `docs/plans/2026-09-29-v6-surfaces-ui.md` | Completed V6 session presentation plan | Shared vocabulary, legacy filter compatibility, independent evidence dimensions and source-coverage disclosures; included in the V6 gates at `1c02db91`. |
 | [2026-09-29-plan-v6-opencode-cost.md](2026-09-29-plan-v6-opencode-cost.md) | `docs/plans/2026-09-29-v6-opencode-cost.md` | Completed OpenCode reported-zero trust plan | Scoped cost work and mandatory core cache handoff accepted; versioned observation semantics are documented in the living usage guide. |

docs/upstream-watch.md

@@ -101,8 +101,10 @@ Every command also takes `--concurrency <1-16>` (default 4) and `--registry <fil
   read succeeded.
 - `record` is what the scheduled workflow runs (see [The ledger](#the-ledger)). It builds a
   ledger commit locally and never pushes. `--dry-run` fires nothing and builds nothing; it makes
-  the same read-only branch and pull request checks and lists each thread that would fire the
-  dispatch routine in `wouldFire` (`[]` when none, and on every run that is not a dry run).
+  the same read-only branch and pull request checks. It lists the first three eligible fixes in
+  `wouldFire` and the remaining eligible fixes in `deferred`, without sleeping. `wouldFire` is
+  `[]` when none qualify and on runs that are not dry runs. A preview assumes successful trigger
+  calls; a real trigger failure can stop the run earlier.
 - `ledger` prints the recorded lines that match every filter given, or the records as JSON.
   `--since` selects by the event's date; `--recorded-since` by when a run recorded it
   (`recordedAt`).
@@ -112,7 +114,10 @@ registry is invalid or the ledger branch cannot be read. `record` exits 3 when b
 registry is invalid, the ledger branch cannot be read or holds a malformed line, or not one
 upstream thread could be read (our own tracking issues do not count). It also exits 3 when the
 ledger commit could not be built; the routine sessions it already fired are then listed in
-`fired` and on stderr, and the next run fires them again. A `--since` in the future is a
+`fired` and on stderr. Its JSON also preserves dispatch errors and the actual `deferred` list.
+Missing ledger evidence can allow a later run to fire again. Blind results consistently include
+empty arrays for unavailable records, errors, previews, deferred fixes and observed sessions.
+An invalid registry takes precedence over a future `--since`; otherwise a future `--since` is a
 command-line error.
 
 The Ruflo support window (the newest six minors, never fewer than those released in the last
@@ -180,6 +185,8 @@ trailer, else seven days ago. A failed read is tried twice more (after 2 and 10
 counts as an error. Replies, acknowledgements, closures and merges count only after that start;
 the other events repeat while their condition holds, dated by the upstream fact, so the same fact
 always gives the same line, and a line already in `events.ndjson` is never recorded again.
+Release state is checked before ledger deduplication, independently of the start of the window.
+Advancing `Checked-At` therefore does not hide a deferred release that remains eligible.
 
 A run with new records makes one commit: the previous records plus the new ones, a message with
 one plain sentence per new record, and the trailer `Checked-At:` with the run's start time, or the
@@ -259,9 +266,10 @@ never posts, pushes or merges without explicit confirmation.
 at 14:17 UTC (`17 14 * * *`) and on demand (`workflow_dispatch`, with `record` off for a dry run in
 the job summary and an optional `since`). GitHub may start a scheduled run late or, under heavy
 load, drop it; the next run's window covers the gap. No model runs in it: the script decides the
-records, the firings and the notice text. Every job summary says how many dispatch routine
-sessions the run would start (`wouldFire`) and lists them, so a dry run shows what a real run
-would fire.
+records, the firings and the notice text. Both preview and record summaries list `wouldFire`
+and `deferred`, count dispatch errors, and report blind failures. A dry run previews at most
+three fixes; a real run leaves `wouldFire` empty. If the ledger build fails after firing, the
+summary also preserves the observed session links.
 
 The `watch` job has `contents: write` (to push the ledger branch and comment on its commit),
 `actions: read` and `pull-requests: read`. Its steps, in order:
@@ -285,22 +293,41 @@ reads neither upstream repositories nor the ledger (a cloud session reaches only
 attached to it). It has no schedule: the watch fires its API trigger
 ([Add an API trigger](https://code.claude.com/docs/en/routines#add-an-api-trigger)) once for each
 `released` line with `branch=` whose branch does not exist yet, sends the line's id, version and
-branch as the payload, and records a `fired` line with the session link. If the branch has not
-appeared three days later it fires once more; after that the job fails and names both sessions.
+branch as the payload, and records a `fired` line after HTTP 200 with a string session URL.
+A run attempts at most three eligible fixes, with 15 seconds between fixes. It stops attempting
+fixes after the first trigger error and lists later eligible fixes in `deferred`. These remain
+eligible for later checks; their execution is not guaranteed, and repeated errors on an earlier
+fix can starve the backlog. PR observation still runs after the cap or a trigger error.
+If the branch has not appeared three days after a recorded firing, the fix becomes eligible
+for one more firing; after two recorded firings the job fails and names both sessions.
 To clear that, create the branch or move the entry's status off `watching` and
 `fixed-unreleased`. Deleting a dispatch branch (for example after closing its pull request) lets
 the watch fire again, within the two firings per thread. A day with nothing to dispatch costs
 nothing in claude.ai.
 
+The [routine API reference](https://platform.claude.com/docs/en/api/claude-code/routines-fire)
+documents retries for HTTP 500 and 503. The watcher allows three attempts per fix, waiting
+2 and then 4 seconds. It does not retry other statuses, thrown network errors or timeouts,
+HTTP 200 without a session URL, or any response already containing a string session URL.
+An error is an observed response, not proof that the server created no session. The API has no
+idempotency key, so retries cannot guarantee exactly one session. Error body messages and
+request IDs redact the configured token before truncation, remove control characters and are
+limited to 200 characters each.
+
+The historical [scheduled run 36583034986](https://github.com/pacphi/agentic-kit/actions/runs/36583034986)
+at revision `94890a00` reported seven HTTP 503 dispatch errors without a parsed string session
+URL. Those logs do not establish that no server-side sessions existed or that this was the
+first dispatching run; the reconciliation tests use injected responses, not real dispatches.
+
 The trigger token is the repository secret `UPSTREAM_DISPATCH_TOKEN`, created in claude.ai; the
 routine id is in the workflow. The routine acts as the maintainer's GitHub user and its session
 has GitHub write tools, so the limits below are instructions in its prompt, not permissions. The
 platform checks each push to a branch not prefixed `claude/` and refuses it when the branch is
 protected, someone else has an open pull request from it, or it carries someone else's commits
 ([Repositories and branch permissions](https://code.claude.com/docs/en/routines#repositories-and-branch-permissions)).
 GitHub does not notify you of your own pull request by default, so the watch checks for an open
-draft pull request while the entry is `watching` or `fixed-unreleased`, for up to seven days after
-its latest firing. When found, it records `dispatch-pr` and its notice says the draft is ready.
+draft pull request while the entry is `watching` or `fixed-unreleased`, for strictly less than seven days after
+its latest firing, and stops after recording `dispatch-pr`. When found, it records `dispatch-pr` and its notice says the draft is ready.
 After that window, a late pull request needs manual reconciliation; the `fired` evidence remains
 in the ledger.
 

scripts/upstream-watch.mjs

@@ -250,7 +250,7 @@ const WEEK = 7 * 86_400_000;
 function blindRecord(error, { stdout, stderr, json }, extra = {}, { writeError = true } = {}) {
   if (writeError) stderr.write(`${error}\n`);
   const result = {
-    blind: true, error, records: [], fetchErrors: [], dispatchErrors: [], wouldFire: [], fired: [], parent: null, commit: null, notice: { post: false, body: '' }, ...extra,
+    blind: true, error, records: [], fetchErrors: [], dispatchErrors: [], wouldFire: [], deferred: [], fired: [], parent: null, commit: null, notice: { post: false, body: '' }, ...extra,
   };
   if (json) stdout.write(`${JSON.stringify(result, null, 2)}\n`);
   return BLIND;
@@ -286,7 +286,7 @@ async function record(registry, fetcher, options, { stdout, stderr, now, ledgerS
   const all = ledgerEvents(report, registry, { since });
   const released = all.filter((event) => event.event === 'released' && event.fields.branch);
   const eligibleIds = new Set(registry.watch.filter((entry) => PENDING.has(entry.status)).map((entry) => entry.id));
-  const fired = await dispatch({ released, records: ledger.records, dispatcher, repo, sentinel, now, recordedAt: runAt, dryRun: options.dryRun, eligibleIds });
+  const fired = await dispatch({ released, records: ledger.records, dispatcher, repo, sentinel, now, recordedAt: runAt, dryRun: options.dryRun, eligibleIds, pause: sleep });
   const records = [...withoutRecorded(all, recorded).map((event) => toRecord(event, runAt)), ...fired.records];
   const checkedAt = fetchErrors.length ? (ledger.checkedAt ?? since) : runAt;
   let commit = null;
@@ -298,20 +298,21 @@ async function record(registry, fetcher, options, { stdout, stderr, now, ledgerS
         sentences: records.map((item) => commitSafe(sentence(item))),
       });
     } catch (error) {
-      // The routine already ran for these; without the commit the next run fires again.
+      // These sessions were observed; without the commit, later runs may fire again.
       const sessions = fired.records.filter((item) => item.event === 'fired');
       for (const item of sessions) stderr.write(`Fired ${item.id} before the ledger commit failed: session ${item.fields.session}\n`);
-      return blindRecord(`Could not build the ledger commit: ${error.message}`, io, { fetchErrors, dispatchErrors: fired.errors, fired: sessions });
+      return blindRecord(`Could not build the ledger commit: ${error.message}`, io, { fetchErrors, dispatchErrors: fired.errors, fired: sessions, deferred: fired.deferred });
     }
   }
   const body = renderNotice({ records, mention, date: runAt.slice(0, 10), recordedAt: runAt });
   const result = {
-    since, sinceSource, checkedAt, blind: false, records, fetchErrors, dispatchErrors: fired.errors, wouldFire: fired.wouldFire,
+    since, sinceSource, checkedAt, blind: false, records, fetchErrors, dispatchErrors: fired.errors, wouldFire: fired.wouldFire, deferred: fired.deferred,
     parent: ledger.commit, commit, notice: { post: Boolean(body), body },
   };
   for (const item of fetchErrors) stderr.write(`Could not check ${item.id}: ${item.error}\n`);
   for (const item of fired.errors) stderr.write(`Dispatch ${item.id}: ${item.error}\n`);
-  const lines = [...records.map((item) => item.line), ...fired.wouldFire.map((item) => `Would fire ${item.id} ${item.version} ${item.branch}`)];
+  const lines = [...records.map((item) => item.line), ...fired.wouldFire.map((item) => `Would fire ${item.id} ${item.version} ${item.branch}`),
+    ...fired.deferred.map((item) => `Deferred for a later check: ${item.id} ${item.version} ${item.branch}`)];
   stdout.write(options.json ? `${JSON.stringify(result, null, 2)}\n` : lines.length ? `${lines.join('\n')}\n` : 'No new records.\n');
   return 0;
 }