Skip to content

limits.md

Latest commit

 

History

History
620 lines (556 loc) · 41.6 KB

limits.md

File metadata and controls

620 lines (556 loc) · 41.6 KB
Raw

Automation Limits

  • Status: active generated/validated configuration reference
  • Owner: ClawSweeper maintainers
  • Source of truth: config/automation-limits.json, Worker overrides, and scripts/check-limits.ts
  • Last verified: openclaw/clawsweeper@647503ec44b8e777dd172adf974a945367da0d19
  • Update when: worker budgets, lane ratios, workflow literals, or production capacity overrides change; run pnpm run check:limits

Read when changing ClawSweeper throughput, Codex fan-out, or repair dispatch capacity.

config/automation-limits.json is the source of truth for the global worker budget. It deliberately has one main global knob, workers.max, because that is the number we normally tune when Codex or GitHub rate limits get tight. Most lane-specific limits are derived from that budget; imported cluster repair has a separate explicit knob so it can stay tightly bounded unless a maintainer intentionally opens it wider. Safety thresholds such as close age floors, apply delays, retry counts, and comment caps stay near the code that owns those decisions.

Review-intake rate limits and the default-off per-author PR-budget and obsolescence policies follow that safety-threshold rule. Their tunables and safety floors live beside the owning policy rather than in the global worker budget:

Environment variable Default Meaning
CLAWSWEEPER_AUTHOR_PR_BUDGET_CLOSE_ENABLED false Enables live per-author budget closes.
CLAWSWEEPER_AUTHOR_PR_BUDGET 15 Allowed open PRs per external author and repository.
CLAWSWEEPER_AUTHOR_PR_BUDGET_MAX_CLOSES_PER_RUN 5 Gradual trim cap per author in one apply run.
CLAWSWEEPER_BULK_FILER_THRESHOLD 10 Recent authored issues not closed as completed.
CLAWSWEEPER_BULK_FILER_WINDOW_DAYS 7 Lookback window ending at the issue's creation.
CLAWSWEEPER_STALE_VERSION_BUG_CLOSE_ENABLED false Enables stale-version bug closes after 120 days.
CLAWSWEEPER_OBSOLETE_FIX_PR_CLOSE_ENABLED false Enables obsolete small-fix PR closes after 90 days.

The bulk-filer count leaves out issues closed as completed. Every other issue in the window counts, including open, rated, not-planned, and duplicate issues. Rating labels and priority labels do not change the count: a rating can still change between reviews of an unchanged issue, so it is not yet a stable signal (#1752).

See author-pr-budget-close-policy.md for the rating, proof, inactivity, engagement, and fail-closed gates. See obsolescence-close-policies.md for the 120/90-day issue and 90/30-day PR age/inactivity contracts.

GitHub repository variables still override selected live limits. When a variable is unset, workflows read the checked-in budget after checkout. Broad manual review uses the same queue limits as scheduled feeds; manual shard/batch controls are retired. pnpm run check:limits verifies the remaining derived Worker, workflow, and documentation values.

The mental model:

  • workers.max is the global Codex capacity budget.
  • GitHub Actions workflows that only route comments, publish exact-review results, or reconcile leases do not execute Codex and do not consume that budget.
  • Priority lanes are repair, issue implementation, and exact-item review.
  • Background lanes are normal review and hot intake.
  • Assist has a small fixed cap because it is lightweight maintainer Q&A, not a derived review or repair lane.
  • Background lanes shrink when priority work is already active.
  • Runtime overrides are escape hatches, not the normal tuning surface.

Worker Budget

Audit hydration is bounded independently of review capacity:

Name Current Meaning
audit.max_parallel_targets 3 Target audits dispatched per wave; wait for every run to complete before the next wave.

The fanout reads this limit through AUTOMATION_LIMITS in src/limits.ts. The signed audit cursor checkpoints remaining targets and outstanding children; resumed batches drain those children before admitting a new wave. It does not alter worker budgets, exact-review admission, or target selection.

Name Current Meaning
workers.max 128 Maximum global Codex worker budget used to derive lane limits.
workers.reserve_for_interactive 16 Worker slots background lanes leave open for exact/manual/urgent work.
workers.expansion_reserve 8 Extra slots background lanes leave open for independently planned matrix expansion.
workers.minimum_background 16 Target floor for background progress when enough global capacity is available.
lanes.exact_review.max_concurrent 80 Maximum concurrent exact-item review workflow runs admitted to Codex.
lanes.exact_review.target_max_concurrent 64 Maximum concurrent exact-item review workflow runs one target repository may consume.
lanes.exact_review.actions_budget 194 Separate Actions budget; preserves publication and control-plane headroom without raising the review ceiling.
lanes.assist.max 10 Maximum concurrent lightweight assist jobs.
lanes.repair.cluster_max_live_runs 2 Default live repair workflow cap for imported gitcrawl cluster dispatches.

Derived Limits

Review limits are percentages of workers.max, with small fixed floors. With workers.max = 128, normal and hot intake have configured ceilings of 89 and 44. The interactive and expansion reserves leave 104 background slots when quiet.

Name Current Meaning
exact_review.concurrent_max 80 Exact-item review admission cap, clamped to workers.max.
exact_review.target_concurrent_max 64 Exact-item per-target admission cap, clamped to global exact-review capacity.
assist.default 10 Maintainer assist job cap.
review_shards.normal_default 89 Quiet-system normal review shard ceiling.
review_shards.normal_active_floor 38 Minimum active normal review shards to keep queued for openclaw/openclaw.
review_shards.hot_intake_default 44 Quiet-system broad hot-intake review shard ceiling.
review_shards.exact_item_default 1 Exact-item hot-intake shard count.
review_shards.hard_cap 128 Maximum accepted review shard count.

Formula summary:

  • normal review: 70% of workers.max
  • normal active floor: 30% of workers.max
  • hot intake: 35% of workers.max
  • review hard caps: workers.max

Dynamic Scheduling

Manual normal review and manual hot intake are background lanes. Before they dispatch, the workflow asks pnpm run workflow -- worker-limit <lane> for the current allowance. Automated normal and hot cycles enqueue exact-review work instead, so the Durable Object's rate, backlog, and claim limits own their concurrency.

The scheduler does this for background lanes:

  1. start with workers.max
  2. subtract active priority work, currently exact-item sweep runs
  3. subtract active background work already known to the workflow, meaning other active normal/hot sweep runs
  4. reserve workers.reserve_for_interactive
  5. reserve workers.expansion_reserve for independently planned matrix waves
  6. cap the result at the lane's derived quiet-system ceiling
  7. return at least 1 so an enabled lane can still make slow progress

The normal result is then reduced when the exact-review queue is under pressure. Each planner reads the public, unauthenticated GET /api/exact-review-queue endpoint once and uses its review-lane pending count and oldest-pending age. The top-level pending, ready, admissible, handoff, and pressure fields are aliases for that review lane; publication health and backlog remain under lanes.publication and never throttle review producers. A failed, timed-out, or malformed response uses the conservative unavailable-pressure budget until the next healthy probe.

Tier Trigger, either condition Background budget
none Below both soft thresholds Normal dynamic budget
soft At least 150 pending or oldest pending is at least 30 minutes ceil(normal dynamic budget * 0.5)
hard At least 400 pending or oldest pending is at least 2 hours max(1, floor(normal dynamic budget * 0.1))

The thresholds can be overridden with repository variables or process environment variables. Values are non-negative counts or millisecond durations; unset, empty, or invalid values use the defaults.

Environment variable Default
CLAWSWEEPER_QUEUE_PRESSURE_SOFT_PENDING 150
CLAWSWEEPER_QUEUE_PRESSURE_HARD_PENDING 400
CLAWSWEEPER_QUEUE_PRESSURE_SOFT_AGE_MS 1800000
CLAWSWEEPER_QUEUE_PRESSURE_HARD_AGE_MS 7200000

Broad manual and scheduled review both use queue-advertised candidate capacity and the shared admission target. Planners serialize by target, offer candidates, and finish without reserving matrix workers. The legacy worker-limit pressure multiplier remains a helper for other callers; it no longer controls a hosted manual review matrix. Repair, assist, issue implementation, cluster repair, and exact-item admission retain their existing priority budgets.

Priority lanes do not subtract the interactive reserve. They cap themselves at their derived lane ceiling and at the remaining global budget after other active priority work.

Exact-item webhooks are admitted by the dashboard Worker's durable ExactReviewQueue, not by a live Actions semaphore. Its production queue logic, including storage migrations, leasing, reclamation, debounce, and shedding, lives in dashboard/exact-review-queue.ts; dashboard/worker.ts is the fetch and dashboard router and only imports that service boundary. The queue coalesces deliveries by repository and item number, so a new webhook updates the latest desired review rather than consuming another runner. Only EXACT_REVIEW_QUEUE_MAX_CONCURRENT leased items may dispatch an exact-review workflow at once; production sets this to 80. EXACT_REVIEW_TARGET_MAX_CONCURRENT bounds how many of those slots one target repository may consume; production sets it to 64 so other target repositories retain 16 global slots during an OpenClaw backlog drain. Exact capacity is consumed only while queue work is pending. As those priority workers start, normal and hot-intake planners count them and reduce their next background wave.

EXACT_REVIEW_ACTIONS_BUDGET is deliberately separate from the 128-slot Codex worker budget. Its production value remains 194: at 80 exact reviews and up to 32 publishers it preserves the 16-slot control-plane reserve plus 66 slots of Actions headroom. Full review admission therefore cannot reduce verdict publication to zero, while repair and broad review derivations remain anchored to workers.max = 128.

EXACT_REVIEW_SCHEDULED_MAX_CONCURRENT caps combined scheduled hot intake and normal backfill at 32 active reviews in production. Both dispatching and leased owners count, using the immutable lease decision while a newer desired revision waits. Lowering the cap does not cancel owners or remove pending work. Organic/manual requests remain eligible when the scheduled cap is full. The same bound governs admission, admissible statistics and next-wake calculation, so held backlog does not cause one-second polling. Public scheduled_feed.max_concurrent and active expose these counts; OpenClaw Bay remains observer-only and needs no new controls.

The public scheduled_feed also reports the admission budget itself:

  • burst and token_balance for the global bucket. The balance is negative while organic debt is outstanding, down to -burst.
  • lanes.hot_intake and lanes.normal_backfill, each with target_rate_per_hour, burst and token_balance.
  • throttle_observed_at and throttle_recovery_at after a review completion has reported a GitHub throttle. Scheduled admission is paused until the recovery time.

lanes.review.shed_reasons_since_reset splits review sheds into backpressure, scheduled_rate and unattributed. Together these show whether scheduled backfill is held back by organic debt, by lane rates, by a throttle pause, or by queue backpressure. They are aggregate numbers with no item, repository or credential detail.

Ordinary automatic PR reviews first observe the live head, title, body, and draft state, then wait for EXACT_REVIEW_INITIAL_PR_QUIET_MS (ten minutes). Meaningful changes restart that quiet window, including changes discovered without a webhook. Comments and label-only activity do not extend it. There is no maximum-wait shortcut: a continuously changing PR waits until it settles. Draft-state changes restart the window; existing draft and oversized-PR gates remain in force. Explicit review requests bypass this initial hold.

Once a PR has a completed canonical review, ordinary webhooks and scheduled hot/normal passes cannot review it again. Request @clawsweeper re-review after finishing changes. The additive exact_review_pull_request_answers table records that fact across restarts, report replacement, archive moves, and queue cleanup; existing verified canonical reports with retained completion provenance seed it on first use. Reports whose old receipts were already pruned remain eligible for one fresh answer. Unverified attempts and stale-source requeues do not consume the first answer. Issues keep their existing cadence.

Fresh webhook work waits for EXACT_REVIEW_DISPATCH_DEBOUNCE_MS (90 seconds by default) so rapid edits and pushes coalesce before dispatch. Repeated pending revisions extend that delay up to EXACT_REVIEW_DISPATCH_DEBOUNCE_MAX_MS (three minutes by default) from the item's first enqueue. A superseding source event immediately revokes the old queue lease and starts a fresh debounce window for the latest revision. The old workflow's tuple heartbeat then returns 409; its review step terminates the local review process instead of using GitHub's run-id-only cancellation API, which cannot safely distinguish a later rerun attempt. The replacement is durably scheduled before the old worker observes the revocation, and review_superseded_total records the terminalized review generation. Recovery events never supersede an existing item; only a fresh source revision can revoke an active review. Duplicate workflow deliveries remain non-cancelling and rely on the lease claim tuple, so they cannot terminate the sole valid owner. An older unclaimed workflow cannot pass the replacement lease tuple and exits before review compute. After setup, generation also checks the full lease tuple synchronously before either ordinary or oversized review starts. Known lease_superseded / lease_not_active conflicts exit as successful no-ops; unverified ownership fails before generation and leaves recovery to the queue. This adds one heartbeat request per generation attempt (with the existing bounded transport retries), not a faster polling loop. Earlier checkout/setup is not interrupted, and supersession after that check still relies on the ordinary-review one-minute heartbeat plus request/termination time. Oversized generation retains its post-generation finalization check rather than an active stop loop. Finalization and publication fences remain unchanged. The startup proof records the isolated workflow/process boundary and its limits. Explicit command work and publication work bypass the delay. The PR quiet window above takes precedence for ordinary automatic PRs.

Items under active churn wait longer. When an item has already completed EXACT_REVIEW_ACTIVE_ITEM_REVIEW_THRESHOLD (2 by default) successful review generations within the trailing EXACT_REVIEW_ACTIVE_ITEM_WINDOW_MS (one hour by default), its next organic source revision (a review-triggering issue or pull request webhook action such as synchronize or edited) waits EXACT_REVIEW_ACTIVE_ITEM_DEBOUNCE_MS (ten minutes by default) from the latest pending revision. Further revisions still coalesce into that one pending entry, capped at EXACT_REVIEW_ACTIVE_ITEM_DEBOUNCE_MAX_MS (15 minutes by default) from the first pending enqueue, so a continuously pushed pull request is still reviewed. The active delay only ever lengthens the ordinary debounce. A superseding event still revokes an in-flight lease immediately; only the next dispatch waits. Explicit commands, publication, scheduled intake, repair follow-ups, and recovery work keep their existing timing, as do items below the threshold. PR opened / ready_for_review intake can observe live state immediately, but ordinary PR dispatch still waits for the initial quiet window above. The queue keeps the completion history in an additive exact_review_queue_review_completions table keyed by item and workflow run, pruned to the window; a store without history behaves as before. The next-wake calculation reads the delayed nextAttemptAt, so a held item does not cause extra polling.

When pending depth reaches EXACT_REVIEW_PENDING_SOFT_LIMIT (600 by default), new recovery and scheduled feed work is shed; this threshold counts review work only, so publication backlog cannot consume review admission capacity. Existing items, webhook events, commands, and publications remain admitted, and so does recovery or source-drift work that still carries a command's context, which also passes a GitHub throttle window. The queue reports shed counts under lanes.review.shed_reasons_since_reset and the rolling flow by backpressure versus scheduled_rate; pre-migration totals remain unattributed. Executed reviews debit a durable 220-review/hour scheduled budget with a 24-item burst. The budget meters started review generations, not admissions or claims: a claim records that its claim generation owes one token, and the workflow's startup ownership check, immediately before Codex generation, sends generation_start: true on the lease heartbeat to pay it. Organic work therefore consumes the budget first, once per run that actually reaches generation. Organic admission itself is free and always admitted. Work that never starts generation is never charged: items superseded or coalesced before claim, dedupes, items completed at the dispatch-time live check without a run, runs that claim and then exit at live-item admission or lose the lease during setup, publication work, and acknowledgement-only finalizers. A retried generation-start heartbeat for the same claim generation is the same execution; a rerun attempt, retry, or requeue that claims and starts again is another execution. A scheduled admission debits its token when admitted, because admission is where scheduled work is gated, and marks the item prepaid so its first started generation is not charged again; later starts of that item are charged like organic ones. Organic debt carries on the global bucket down to minus the burst, so scheduled work is admitted only after that bounded debt is repaid. Further organic debits at the floor are forgotten, so this is not a total-work or spend cap. Scheduled work fills the remainder through two lane buckets on top of that global bucket: EXACT_REVIEW_HOT_INTAKE_RATE_PER_HOUR caps hot intake (production sets 30), and normal backfill receives the total minus hot, so frequent hot offers cannot take the remainder that oldest-first backfill needs. The burst splits 35% hot and 65% normal (8 and 16 in production). Re-offering an item that is already pending, dispatching, or leased is a semantic dedupe: it does not advance the queue revision, revoke a lease, or count as new work.

A source-drift loop breaker bounds automatic re-review. The queue counts consecutive source_drift_requeue review generations per item in an additive exact_review_queue_source_drift_loops table, because each completed generation deletes the queue item. Once an item has used EXACT_REVIEW_SOURCE_DRIFT_REQUEUE_LIMIT (production: 3) such generations, the next source-drift requeue is not admitted: the item is parked with reason source_drift_loop, and later automatic requeues are deduped against it with dedupe_scope: source_drift_loop. The publisher still receives a successful dedupe, so the parked item is the only new state. The direct and batch publishers both send that requeue as publisher-source-drift:<producer run id>:<attempt>, so one review generation is requeued at most once. A direct-publication completion that owes a lifecycle requeue (direct_lifecycle_requeue) creates the same automatic generation without /enqueue, so it counts and parks the same way: below the limit its successor revision is pending; at the limit the successor revision is parked with reason source_drift_loop, while the completed revision keeps its requeue lifecycle fact. An organic webhook item action (opened, reopened, edited, synchronize, ready_for_review, converted_to_draft, unlocked, unlabeled), an explicit command, a manual review, or an operator recover-fresh both resets the counter and admits the item normally. A scheduled offer whose sourceUpdatedAt is later than the park releases it for one review generation but does not reset the counter: ClawSweeper's own post-review writes, such as lease-comment cleanup and label syncs, move updated_at and feed hot intake. If that review ends in another source-drift requeue, the item re-parks immediately. Older scheduled offers dedupe with reason source_drift_loop. A source-drift requeue that still carries command context continues that command's status lifecycle, so it is neither counted nor parked. The parked item spends no review capacity, appears under parked_reasons.source_drift_loop, is operator-listable through the parked-review routes, and still receives the five-minute terminal check that removes closed or advanced targets. Set the limit to 0 to disable the breaker. A counter idle for seven days expires: the admission lookup treats it as absent before the limit check, and parking never refreshes an expired row.

Deterministic no-op holds stop the other observed self-feeding shape without a counter. When a review run completes successfully on the guarded locked-conversation path, the oversized PR path, or the maintainer-request guard, sweep.yml sends review_hold (locked_conversation, oversized_pull_request, or maintainer_request_required) with the lease completion. Without command context, without input that arrived during the lease, and (for oversized PRs) with a pinned head, the queue keeps the row parked under that reason instead of deleting it. Scheduled offers then dedupe with that dedupe_reason, automatic recoveries dedupe with dedupe_scope: review_hold, and neither claims a run, so neither debits the scheduled budget nor counts toward review_runaway_health. Organic events, commands, manual reviews, the parked terminal check, and parked-review fresh recovery release it exactly as for source_drift_loop. The Worker must recognize a hold reason before a workflow sends it; deploy the Worker first when adding a reason. Older workflows that omit the field keep the previous behavior.

The queue also applies the maintainer-request hold before dispatch, so the common case starts no runner. The live item check that runs before every dispatch evaluates the same rule for automatic work (no command, manual request or publication). GitHub shows a private org member to the App as CONTRIBUTOR, so for a CONTRIBUTOR author the queue and the webhook gate read the author's repository permission (admin or maintain counts as a maintainer, as in the runner admission check), cached per author for one hour. A held item parks under maintainer_request_required exactly as after a skipped run. A failed permission read dispatches as before, and the runner check still applies.

The same queue keeps a trailing 24-hour history of claimed review runs per item (exact_review_queue_review_generations, keyed by item, run ID, and attempt; publication and finalizer-only claims are excluded). An item with more than EXACT_REVIEW_RUNAWAY_REVIEWS_PER_DAY (production: 24) claimed reviews in that window is a runaway. review_runaway_health reports degraded with reason review_runaway, the runaway count, and up to five verified-public sample keys; the dashboard health summary raises it amber. This alert observes; it never changes admission.

Exact-review result publication has a separate adaptive Actions lane. Source fallback minimum, base, and maximum are 4, 24, and 48; production overrides them to 8, 32, and 32, independently of the larger review ceiling. Canonical publication batches retain eight slots. The controller records GitHub pressure, cooldown, recovery, and demand telemetry, and can scale within that production range. Admission enforces a 16-slot control-plane reserve inside EXACT_REVIEW_ACTIONS_BUDGET; with 80 active exact reviews and the current production maximum of 32 publisher slots, another 66 slots remain as configuration headroom rather than protected reserve. Newly accepted, published batch members contribute one recovery success each after cooldown, only when publication completes without requeueing. Replays, superseded members, and requeued revisions earn no credit. A batch carrying quota observations earns no positive credit, so its negative feedback takes precedence. Its checkout, artifact handling, comment sync, and result routing are deterministic control-plane work: they consume GitHub runners, but not Codex slots. The comment router and the singleton lease reconciler follow the same accounting rule. Dashboard Codex capacity therefore counts only jobs whose steps execute Codex and does not deduct these control-plane workflows from workers.max.

Legacy state-repository publication once limited exact-review preparation to four concurrent size-8 batches. Canonical Worker publication removes that shared Git writer constraint, so production now admits 8 preparation batches (up to 64 publication members) while retaining the Durable Object's transactional SQLite ownership boundaries.

Each batch now prepares at most two members concurrently. This changes only per-batch preparation: publication maximum 32, eight batch owners, review and worker budgets, and intake limits are unchanged. A quota observation still defers later members without charging their retry budgets; at most the two already-started members may have requests in flight. The controlled trial and rollback criteria are in backlog recovery proof.

The durable control plane owns reset-aware GitHub credential circuits for both review authority reads and publication batching. The repository Actions pool is identified only as actions:openclaw/clawsweeper; target App pools are identified by the non-secret target owner. A primary or secondary rate-limit observation persists its reset deadline in the queue dispatcher and suppresses new batch dispatch for only the affected pool. Each matching member resumes at its reset-plus-deterministic-jitter boundary, and the next alarm uses the earliest pending member recovery boundary. A later observation may extend but cannot shorten an open circuit. When gh omits response headers, the publisher performs at most one same-credential /rate_limit lookup; the one-minute fallback is used only when no authoritative reset is available.

Legacy exact-review payloads that do not carry target_branch are durably held before admission while the target App resolves the repository default branch; they never silently default to main. The same owner-scoped target App circuit protects that lookup and direct-webhook pull-request head verification. The first reset-aware 403/429 opens the circuit; later reservations for that owner defer to the shared deadline without another GitHub request or another reservation attempt, while other owners continue. A carried valid branch takes the normal enqueue path with no branch lookup. The status payload exposes these pre-admission records under lanes.review.authority_pending; owner-scoped circuit affected_pending counts include them, so quota pressure before normal queue admission remains visible.

The first quota failure in a preparation batch stops later members before artifact download. The observing member records a normal retryable failure; members that never made a GitHub request are requeued after the shared reset with deterministic 1-30 second per-member recovery jitter and do not consume their individual publication retry or dead-letter budgets. A classifier-approved public openclaw/openclaw read may make one bounded fallback from the exhausted Actions token to the already-authorized target App token. That success does not close the Actions circuit, and mutations, private reads, and unsupported routes never use the repository token.

lanes.publication.github_request_metrics measures the successful and failed request budget by redacted credential scope, endpoint category, public-read versus App-operation class, first versus repeated revision, and outcome. It contains counts only: no repository item number, URL, content, token, token hash, authorization header, or filesystem path is retained. Use these counters to justify later cache or metadata-read consolidation; live freshness and pre-mutation guards remain mandatory.

Each dispatched workflow claims its opaque lease before checkout. Protocol v2 binds claim and completion to the item key, lease revision, run attempt, claim generation, source head (for pull requests), and an immutable decision snapshot. For pull requests, the review-start reservation rejects a claimed source head that no longer matches the live head, then heartbeats that full queue tuple before deleting any different-revision placeholder. During the rolling-upgrade window, dispatches nest the strict tuple under queue_claim, also carry the immutable v1 snapshot, and the Worker accepts lease-id-only finalization only for claims recorded as protocol v1. Duplicate dispatches and stale workflows cannot claim the same lease, and a completion immediately schedules a known newer revision. Failed and cancelled executors requeue their item with bounded retry backoff. Successful finalizer reports stay leased until a signed terminal-run reconciliation backstop confirms that exact GitHub attempt completed successfully; this backstop can also recover terminal failed or cancelled runs before lease expiry. Event-triggered lease repair shares one running and one pending job in exact-review-reconcile-workflow-run with cancel-in-progress: false. A five-minute successful-reconciliation guard suppresses redundant scans; skipped and observer-only runs do not reset it. Per-run reliability observations retain their own jobs so no telemetry is lost. The independent */15 sweep reads /internal/exact-review/claimed-runs, checks exact GitHub attempts in batches of eight, and posts terminal tuples to /internal/exact-review/reconcile. It bypasses the event cooldown, providing an approximately 15-minute backstop for coalesced terminal events, subject to Actions scheduling and API availability. See Scheduler for the bounded Actions history reads that implement the cooldown. Unclaimed dispatches expire after six minutes and receive a new opaque lease; delayed workflows holding the expired lease cannot claim it. The six-minute timer covers only the GitHub dispatch-to-claim handoff. Once the first workflow step claims the item, the 130-minute execution lease and one-minute heartbeats take over, so the handoff recycler cannot race a normal four-minute review completion. Run-attempt binding and a per-claim generation check keep delayed terminal decisions from releasing a later rerun; queued and in-progress runs are never released. If a workflow never claims or completes, the Durable Object reclaims the expired lease. Claimed review workers heartbeat every minute; after the first heartbeat, EXACT_REVIEW_HEARTBEAT_GRACE_MS bounds liveness to 20 minutes by default while never extending the original 130-minute execution lease. Leases created before heartbeat support was deployed retain their original execution expiry. This keeps capacity waiting and retry state out of GitHub Actions runners.

Publication completion distinguishes durable publishes from results superseded by a newer or closed remote tuple. Superseded publications terminate without a GitHub mutation and do not count as successful publishes. GitHub/transient failures retry for at most 12 attempts or 24 hours; deterministic permanent failures receive two confirmation retries before entering the bounded dead-letter store. An artifact unavailable for three attempts atomically queues one fresh review. Public queue status reports publish, supersede, retry, refresh, and dead-letter totals separately; signed internal endpoints list, replay, resolve, and exact-revision supersede records without exposing decisions publicly.

Examples with the current config:

  • Quiet system: manual normal review can request 89 shards, with 104 background slots available after reserving 16 for interactive work and 8 for matrix expansion. When the queue capacity probe is unavailable, a single-target scheduled plan falls back to offering up to 50 candidates to the durable 220-review/hour admission target with a 24-item burst instead of starting matrix shards.
  • Four active repair workers and 96 active background workers: normal review gets four because 128 - 16 interactive reserve - 8 expansion reserve - 4 priority - 96 background = 4.

Use these commands to inspect the effective values from a checkout:

pnpm run --silent workflow -- worker-config
pnpm run --silent workflow -- limit review_shards.normal_default
pnpm run --silent workflow -- worker-limit normal_review

Change workers.max first when tuning review-side rate-limit pressure. For example, setting workers.max to 40 automatically makes normal review 28 and hot intake 14. Existing repair lanes keep their 40% derived caps, while imported cluster repair remains separately bounded until lanes.repair.cluster_max_live_runs is raised.

Canonical record export

/internal/state/records/export defaults to 200 records and accepts at most 200 per request. Its reconstruction cap is also 200, and the hydration client requests 200 by default. A replay of 111,000 small records therefore needs 555 requests instead of 2,220 with the former 50-record cap.

The source-content budget remains 2 MiB and the serialized-entry budget remains 4 MiB. Byte preflight selects identities before a single batched source read; chunked records retain their existing reassembly. Large records can end a page before the count limit, and nextCursor advances only through returned records. The existing single-record progress exception is retained: the first record is returned even if it exceeds a page byte budget, preventing an unexportable record from pinning the cursor. The serialized budget measures entries, not the outer JSON envelope.

These limits are owned by dashboard/exact-review-queue.ts, implemented in dashboard/exact-review-direct-publication.ts, and consumed by scripts/worker-records.ts. Regression coverage lives in test/record-export-bounds.test.ts and test/worker-records-request.test.ts.

Runtime Overrides

  • EXACT_REVIEW_HOSTED_TARGET_ADMISSION_FRESH_MS: intake's no-probe public-admission cache window (default 60,000 ms).
  • EXACT_REVIEW_HOSTED_TARGET_ADMISSION_MAX_STALE_MS: intake fallback after retryable probes (default 1,800,000 ms; zero disables caching; terminal probes revoke).
  • EXACT_REVIEW_STALE_PUBLICATION_PRUNE_LIMIT: stale-revision cleanup per alarm (default 100; zero disables).
  • EXACT_REVIEW_PUBLICATION_RECOVERY_SUCCESSES overrides how many consecutive clean publications raise the adaptive publication ceiling by one step; the default is 10 (clamped 1-1000). The former hardcoded 50 pinned the lane at its minimum under hourly rate-limit bursts because the counter resets on every failure.
  • CLAWSWEEPER_IDEA_REVIVAL_REACTIONS retains the positive-reaction threshold that prevents closing an old feature request with community traction; the default is 5. It no longer configures an automatic reopening watcher.
  • REVIEW_PLACEHOLDER_MAX_CHECKS overrides the number of search candidates examined per open and closed state class by each 15-minute orphaned-placeholder recovery pass; the default is 20 and the maximum is 1000. Independent durable cursors rotate both state classes so non-actionable matches cannot permanently pin later placeholders behind the bounded pass.
  • REVIEW_PLACEHOLDER_MIN_AGE_HOURS overrides how old the latest ClawSweeper bot review-start placeholder must be before recovery; the default is 2 hours and the maximum is 720 hours.
  • REVIEW_PLACEHOLDER_MAX_RECOVERIES overrides the number of orphaned review placeholders enqueued per recovery pass; the default is 5 and the maximum is 100.
  • Placeholder recovery logs GitHub Search matched and derived remaining counts for backlog telemetry. Search counts can lag comment mutations, so only failed cleanup or enqueue actions for live-verified orphans make the scheduled run fail.
  • EXACT_REVIEW_DISPATCH_DEBOUNCE_MS overrides the 90,000 ms coalescing delay for fresh non-command exact-review events.
  • EXACT_REVIEW_DISPATCH_DEBOUNCE_MAX_MS overrides the 180,000 ms maximum coalescing window measured from the item's first enqueue.
  • EXACT_REVIEW_ACTIVE_ITEM_REVIEW_THRESHOLD overrides the two completed review generations (clamped to 1-100) that mark an item as under active churn.
  • EXACT_REVIEW_ACTIVE_ITEM_WINDOW_MS overrides the 3,600,000 ms trailing window for counting those completions (at most 24 hours); 0 disables the active-item debounce.
  • EXACT_REVIEW_ACTIVE_ITEM_DEBOUNCE_MS overrides the 600,000 ms delay from the latest organic revision of an active item (at most one hour).
  • EXACT_REVIEW_ACTIVE_ITEM_DEBOUNCE_MAX_MS overrides the 900,000 ms cap for that delay, measured from the first pending enqueue (at most one hour).
  • EXACT_REVIEW_PENDING_SOFT_LIMIT overrides the pending-depth threshold for shedding new recovery and scheduled exact-review work; production sets it to 600.
  • EXACT_REVIEW_SOURCE_DRIFT_REQUEUE_LIMIT sets how many consecutive automatic source-drift review generations an item may use before it parks as source_drift_loop; the default and production value is 3, 0 disables the breaker, and the maximum is 100.
  • EXACT_REVIEW_RUNAWAY_REVIEWS_PER_DAY sets the claimed-review count per item in a trailing 24 hours above which queue health reports review_runaway; the default and production value is 24.
  • EXACT_REVIEW_TARGET_RATE_PER_HOUR sets the fleet-wide scheduled admission refill target; the source fallback is 60 and production sets 220. Claimed organic review executions consume it first and may carry debt down to minus the burst; scheduled work fills only the remainder. Unconditional organic work may exceed the target.
  • EXACT_REVIEW_TARGET_BURST bounds the admission burst and the organic debt floor; the source fallback is six and production sets 24, split 35/65 between the hot-intake and normal-backfill lane buckets.
  • EXACT_REVIEW_HOT_INTAKE_RATE_PER_HOUR caps the hot-intake lane rate (clamped to at least one and below the total); normal backfill receives the total minus hot. When unset or invalid, hot intake keeps 35% of the total. Production sets 30.
  • EXACT_REVIEW_SCHEDULED_MAX_CONCURRENT caps active scheduled review owners; the source fallback is eight and production sets 32, clamped to the global review capacity.
  • Scheduled planners subtract active and pending review work from the 80-slot review capacity before selecting candidates. Target fanout gives every cursor-selected repository a one-candidate floor, then apportions the remaining free capacity by untracked backlog.
  • EXACT_REVIEW_HEARTBEAT_GRACE_MS overrides the 1,200,000 ms exact-review worker heartbeat grace. It is clamped to at least 420,000 ms so a configured grace can never dip near the one-minute worker heartbeat interval during scheduler or network stalls.
  • Broad review-plan.yml dispatches use queue-advertised candidate capacity, bounded by review_shards.hard_cap; there is no per-run shard override.