Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/workflows/upstream-watch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ jobs:
set -e
{
echo "## Upstream watch preview (exit $code)"
jq -r '"since \(.since) (\(.sinceSource)), new records \(.records | length), could not check \(.fetchErrors | length), blind \(.blind), notice \(.notice.post), would fire \(.wouldFire | length)"' watch.json
jq -r 'if .blind then "blind \(.blind): \(.error // "unknown error"), could not check \(.fetchErrors | length)" else "since \(.since) (\(.sinceSource)), new records \(.records | length), could not check \(.fetchErrors | length), blind \(.blind), notice \(.notice.post), would fire \(.wouldFire | length)" end' watch.json
jq -r '(.wouldFire // [])[] | "- would fire \(.id) \(.version) \(.branch)"' watch.json
echo; echo '```text'; cat errors.txt; echo '```'
} >> "$GITHUB_STEP_SUMMARY"
Expand Down Expand Up @@ -102,7 +102,7 @@ jobs:
set -e
{
echo "## Upstream watch (exit $code)"
jq -r '"since \(.since) (\(.sinceSource)), new records \(.records | length), could not check \(.fetchErrors | length), dispatch errors \(.dispatchErrors | length), blind \(.blind), commit \(.commit // "none"), would fire \(.wouldFire | length)"' watch.json
jq -r 'if .blind then "blind \(.blind): \(.error // "unknown error"), could not check \(.fetchErrors | length), dispatch errors \(.dispatchErrors | length)" else "since \(.since) (\(.sinceSource)), new records \(.records | length), could not check \(.fetchErrors | length), dispatch errors \(.dispatchErrors | length), blind \(.blind), commit \(.commit // "none"), would fire \(.wouldFire | length)" end' watch.json
jq -r '(.wouldFire // [])[] | "- would fire \(.id) \(.version) \(.branch)"' watch.json
echo; echo '```text'; cat errors.txt; echo '```'
jq -r '.notice.body' watch.json
Expand All @@ -120,7 +120,8 @@ jobs:
if: env.RECORD == 'true'
run: |
[ "$(jq -r '.notice.post' watch.json)" = true ] || { echo 'Nothing needs the maintainer.'; exit 0; }
commit=$(jq -r '.commit' watch.json)
commit=$(jq -r '.commit // empty' watch.json)
[ -n "$commit" ] || { echo 'Notice requested without a ledger commit.' >&2; exit 1; }
jq -r '.notice.body' watch.json > notice.md
test -s notice.md
gh api "repos/$GITHUB_REPOSITORY/commits/$commit/comments" -F body=@notice.md --jq .html_url
Expand Down
30 changes: 30 additions & 0 deletions docs/archive/2026-09-29-plan-upstream-watch-followups.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Upstream watch followups

## Status at archival

Implemented and independently reviewed on `fix/upstream-watch-followups` at
`b75c1e3fec4f645700f0f6d18220956867b7e5d0`. All eight local gates passed at that
source: unit/legacy tests, browser UI, typecheck, lint, complexity, Markdown,
build, and offline links. Controller workflow commits `aebe2ae6` and `b75c1e3f`
complete items 11/12; their shell and jq behavior was independently exercised.
Green `develop@84307574` was subsequently merged at `ff1eff27` without conflicts.
Final-head PR CI and merge remain pending at capture; no real dispatch or
notification was performed for this lane.

M7 distinguishes deterministic validation failures from transient retries.
M8 bounds eligible fired-PR polling to seven days after the latest firing,
retaining the ledger; a later PR requires manual reconciliation. All twelve
minors have a fix, regression proof of existing behavior, or explicit no-change
disposition. M10 remains declined. Current behavior is documented in
[Upstream watch](../upstream-watch.md).

## Execution and acceptance

Scope: M7, M8, and the twelve numbered deferred minors in the recovered PR #253 report. M10 is declined because the numeric PR field matches emitted records. This lane owns watcher scripts, focused tests, one Codex registry entry, and current watcher documentation. The integration owner owns workflow edits.

1. Add focused failing tests for deterministic retry failures, bounded dispatch polling, ledger errors, and the numbered edge cases. Keep synthetic fetch, dispatch, and git boundaries.
2. Implement the smallest watcher changes that make those tests pass. Check already-correct behavior and record no-change findings without empty commits.
3. Commit independently verifiable items separately. Run focused Node tests and static checks without a full suite, dispatch call, or GitHub write.
4. Put item 11/12 workflow hunks, commands, per-item dispositions, and residual limits in the ignored C4 report. Stop for independent review.

Acceptance: deterministic errors do not sleep; transient errors retry at most twice; fired PR polling stops after seven days or when registry state is ineligible; invalid ledger registry exits nonzero; notices retain their body and maximum length semantics; all twelve minors are either fixed, proved already handled, or handed off as exact workflow hunks.
1 change: 1 addition & 0 deletions docs/archive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ reconfirmed by this metadata audit. The per-file inventory and limitations are r

| File | Original location | What it was | Why it's historical |
|---|---|---|---|
| [2026-09-29-plan-upstream-watch-followups.md](2026-09-29-plan-upstream-watch-followups.md) | `docs/plans/2026-09-29-upstream-watch-followups.md` | V4 C4 execution plan for bounded PR polling, deterministic retry failures and twelve deferred watcher minors. | Implementation and independent review complete; all eight local gates passed at `b75c1e3f`. Final PR CI and merge were pending at archival. Current contract: [Upstream watch](../upstream-watch.md). |
| [2026-06-upstream-findings-f1-f6.md](2026-06-upstream-findings-f1-f6.md) | `docs/upstream/ruflo-self-improvement-findings.md` | The F1–F6 findings series: proofs/refutations of ruflo's self-improvement claims (Q-learning persistence, state-encoder collapse, SONA learn→inference wiring, native-training misreporting), with filed upstream issues. | Every finding is now fixed upstream: F2 in 3.10.6 ([#2222](https://github.com/ruvnet/ruflo/issues/2222)), F2b in 3.10.7, F3 in 3.10.11 ([#2239](https://github.com/ruvnet/ruflo/issues/2239)), F4 in `@ruvector/ruvllm` 2.5.6 ([RuVector#519](https://github.com/ruvnet/RuVector/issues/519)), F6 in 3.18.1/3.19.0 + ruvllm 2.5.7 ([#2549](https://github.com/ruvnet/ruflo/issues/2549), closed 2026-07-03). |
| [2026-06-token-consumption-incident.md](2026-06-token-consumption-incident.md) | `docs/usage/token-consumption-findings-and-mitigation-2026-06.md` | Root-cause report for the June 2026 token-burn incident: six immortal auto-started daemons consumed ~8.1B tokens over 7 days via headless worker sessions. Produced the opt-in daemon policy, TTL reaper, ⚙ statusline alarm, and `ruflo-token-audit`. | The root cause was fixed upstream in ruflo 3.27/3.28 ([#2661](https://github.com/ruvnet/ruflo/issues/2661)): AI workers are opt-in, launches are governed by a machine-wide budget with telemetry, one supervisor daemon per repo, native daemon TTL. The kit's daemon policy flipped back to default-on (local-only workers) on that baseline; the reapers and token-audit remain as an independent check. |
| [2026-06-11-token-consumption-recurrence.md](2026-06-11-token-consumption-recurrence.md) | `docs/usage/token-consumption-recurrence-and-cleanup-2026-06-11.md` | Follow-up audit 10 days later: 17 daemons had accumulated but the TTL auto-reaper had already contained them; cleanup of daemon-state files, logs, and two plugin MCP servers. | Same incident class as above — governed upstream since 3.27/3.28. Kept as evidence the TTL-reaper safety net worked. |
Expand Down
11 changes: 7 additions & 4 deletions docs/upstream-watch.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,8 @@ Every command also takes `--concurrency <1-16>` (default 4) and `--registry <fil
`--since` selects by the event's date; `--recorded-since` by when a run recorded it
(`recordedAt`).

`report`, `check` and `ledger` exit 0 unless the command line is wrong (2) or, for `ledger`, the
ledger branch cannot be read (3). `record` exits 3 when blind: `gh` cannot reach GitHub, the
`report` and `check` exit 0 unless the command line is wrong (2). `ledger` exits 3 when the
registry is invalid or the ledger branch cannot be read. `record` exits 3 when blind: `gh` cannot reach GitHub, the
registry is invalid, the ledger branch cannot be read or holds a malformed line, or not one
upstream thread could be read (our own tracking issues do not count). It also exits 3 when the
ledger commit could not be built; the routine sessions it already fired are then listed in
Expand Down Expand Up @@ -298,8 +298,11 @@ has GitHub write tools, so the limits below are instructions in its prompt, not
platform checks each push to a branch not prefixed `claude/` and refuses it when the branch is
protected, someone else has an open pull request from it, or it carries someone else's commits
([Repositories and branch permissions](https://code.claude.com/docs/en/routines#repositories-and-branch-permissions)).
GitHub does not notify you of your own pull request by default, so the watch's next run records
`dispatch-pr` and its notice says the draft is ready.
GitHub does not notify you of your own pull request by default, so the watch checks for an open
draft pull request while the entry is `watching` or `fixed-unreleased`, for up to seven days after
its latest firing. When found, it records `dispatch-pr` and its notice says the draft is ready.
After that window, a late pull request needs manual reconciliation; the `fired` evidence remains
in the ledger.

- **Trigger:** API only.
- **Prompt:**
Expand Down
16 changes: 10 additions & 6 deletions scripts/upstream-watch.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import {
import { createDispatcher, dispatch } from './upstream-watch/dispatch.mjs';
import { createFetcher, mapLimit, retrying } from './upstream-watch/fetch.mjs';
import { createLedgerStore, toRecord } from './upstream-watch/ledger-branch.mjs';
import { commitSafe, isoSeconds, renderNotice, sentence } from './upstream-watch/ledger.mjs';
import { LEDGER_EVENTS, commitSafe, isoSeconds, renderNotice, sentence } from './upstream-watch/ledger.mjs';
import { renderEvents, renderReport } from './upstream-watch/render.mjs';

const USAGE = `usage: node scripts/upstream-watch.mjs report [--json] [--concurrency <1-16>] [--registry <file>]
Expand All @@ -28,7 +28,6 @@ const USAGE = `usage: node scripts/upstream-watch.mjs report [--json] [--concurr
const PENDING = new Set(['watching', 'fixed-unreleased']);
// record exits BLIND when gh, the registry, the ledger branch or every upstream thread is unreadable.
const BLIND = 3;
const LEDGER_EVENTS = ['reply', 'acknowledged', 'closed', 'merged', 'released', 'reopened', 'stale', 'retire-proposed', 'retest-due', 'idle', 'fired', 'dispatch-pr'];

class UsageError extends Error {}

Expand Down Expand Up @@ -248,8 +247,8 @@ async function ledgerQuery(registry, options, { stdout, stderr, now, ledgerStore

const WEEK = 7 * 86_400_000;

function blindRecord(error, { stdout, stderr, json }, extra = {}) {
stderr.write(`${error}\n`);
function blindRecord(error, { stdout, stderr, json }, extra = {}, { writeError = true } = {}) {
if (writeError) stderr.write(`${error}\n`);
const result = {
blind: true, error, records: [], fetchErrors: [], dispatchErrors: [], wouldFire: [], fired: [], parent: null, commit: null, notice: { post: false, body: '' }, ...extra,
};
Expand Down Expand Up @@ -286,7 +285,8 @@ async function record(registry, fetcher, options, { stdout, stderr, now, ledgerS
const recorded = ledger.records.map((item) => item.line).join('\n');
const all = ledgerEvents(report, registry, { since });
const released = all.filter((event) => event.event === 'released' && event.fields.branch);
const fired = await dispatch({ released, records: ledger.records, dispatcher, repo, sentinel, now, recordedAt: runAt, dryRun: options.dryRun });
const eligibleIds = new Set(registry.watch.filter((entry) => PENDING.has(entry.status)).map((entry) => entry.id));
const fired = await dispatch({ released, records: ledger.records, dispatcher, repo, sentinel, now, recordedAt: runAt, dryRun: options.dryRun, eligibleIds });
const records = [...withoutRecorded(all, recorded).map((event) => toRecord(event, runAt)), ...fired.records];
const checkedAt = fetchErrors.length ? (ledger.checkedAt ?? since) : runAt;
let commit = null;
Expand Down Expand Up @@ -333,7 +333,11 @@ export async function main(argv, {
stderr.write(`upstream registry is ${registry.registryStatus ?? registry.status}:\n${registry.errors.map((error) => ` ${error}`).join('\n')}\n`);
const status = { status: registry.registryStatus ?? registry.status, errors: registry.errors };
if (options.command === 'record') {
return blindRecord(`upstream registry is ${status.status}`, { stdout, stderr, json: options.json }, { registry: status });
return blindRecord(`upstream registry is ${status.status}`, { stdout, stderr, json: options.json }, { registry: status }, { writeError: false });
}
if (options.command === 'ledger') {
if (options.json) stdout.write(`${JSON.stringify({ registry: status }, null, 2)}\n`);
return BLIND;
}
stdout.write(options.json ? `${JSON.stringify({ registry: status }, null, 2)}\n` : 'No report: the upstream registry is not valid.\n');
return 0;
Expand Down
13 changes: 11 additions & 2 deletions scripts/upstream-watch/dispatch.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { toRecord } from './ledger-branch.mjs';
export const FIRE_URL = (routine) => `https://api.anthropic.com/v1/claude_code/routines/${routine}/fire`;
export const FIRE_HEADERS = { 'anthropic-beta': 'experimental-cc-routine-2026-04-01', 'anthropic-version': '2023-06-01', 'content-type': 'application/json' };
export const REFIRE_AFTER_DAYS = 3;
export const PR_OBSERVE_DAYS = 7;
export const MAX_FIRES = 2;
export const FIRE_TIMEOUT_MS = 30_000;
// `gh pr list --head` matches the branch name in any fork; only a pull request
Expand All @@ -19,6 +20,11 @@ const ROUTINE = /^trig_[A-Za-z0-9]+$/;
const DISPATCH_BRANCH = /^upstream\/[a-z0-9._-]+$/;
const DAY = 86_400_000;

export function sessionList(sessions) {
if (sessions.length < 3) return sessions.join(' and ');
return `${sessions.slice(0, -1).join(', ')}, and ${sessions.at(-1)}`;
}

export function createDispatcher({ exec = run, fetchImpl = globalThis.fetch, env = process.env } = {}) {
return {
async branchExists(branch) {
Expand Down Expand Up @@ -56,7 +62,7 @@ export function createDispatcher({ exec = run, fetchImpl = globalThis.fetch, env
}

/** Fire (or, in a dry run, list in `wouldFire`) each released fix; the branch and pull request lookups only read. */
export async function dispatch({ released, records, dispatcher, repo, sentinel, now, recordedAt, dryRun = false }) {
export async function dispatch({ released, records, dispatcher, repo, sentinel, now, recordedAt, dryRun = false, eligibleIds = null }) {
const out = [];
const errors = [];
const wouldFire = [];
Expand All @@ -68,7 +74,7 @@ export async function dispatch({ released, records, dispatcher, repo, sentinel,
if (await dispatcher.branchExists(branch)) continue;
const firings = recordsOf(event.id, 'fired');
if (firings.length >= MAX_FIRES) {
errors.push({ id: event.id, error: `dispatch did not complete after ${firings.length} firings; see ${firings.map((item) => item.fields.session).join(' and ')}` });
errors.push({ id: event.id, error: `dispatch did not complete after ${firings.length} firings; see ${sessionList(firings.map((item) => item.fields.session))}` });
continue;
}
const newest = Math.max(...firings.map((item) => Date.parse(item.recordedAt)), 0);
Expand All @@ -85,6 +91,9 @@ export async function dispatch({ released, records, dispatcher, repo, sentinel,
}
for (const id of new Set(records.filter((item) => item.event === 'fired').map((item) => item.id))) {
if (recordsOf(id, 'dispatch-pr').length) continue;
if (eligibleIds && !eligibleIds.has(id)) continue;
const latestFiring = Math.max(...recordsOf(id, 'fired').map((item) => Date.parse(item.recordedAt)));
if (!Number.isFinite(latestFiring) || now.getTime() - latestFiring >= PR_OBSERVE_DAYS * DAY) continue;
try {
const branch = recordsOf(id, 'fired').at(-1).fields.branch;
const pr = await dispatcher.openPullRequest(repo, branch);
Expand Down
Loading
Loading