Skip to content

A repo skipped for a held lock is recorded as an error, firing a false Discord FAILED alert and consuming its cycle slot #152

Description

@dmccoystephenson

Summary

When tend (or align) skips a repo because another gardener process holds its per-repo lock, the skip is recorded as outcome="error". Two consequences follow that a lock skip should not have:

  1. A Discord alert titled gardener tend: FAILED — <repo> at Level.ERROR is sent, for a repo where nothing failed.
  2. Under the random/issue-count strategies, the repo is persisted into the cycle's attempted list, so it is not retried until the next full cycle through the garden.

A held lock is the exclusion mechanism working as designed — repo_lock.py's own docstring names two overlapping overnight runs as a case it exists to handle, and describes a locked repo as "a skip-this-run condition ... not a wait". It is not an error condition and should not be alerted or accounted for as one.

Observed

Two overnight runs overlapped on 2026-09-05 (a missed Windows Task Scheduler nightly fired its catch-up shortly after WSL booted, ~50s before a run was started by hand). Stephenson-Software/SimpleAccountRegistry was skipped by the second run, and this was logged:

gardener: Stephenson-Software/SimpleAccountRegistry is already being worked on by another gardener process (its per-repo lock is held) — skipping rather than risking concurrent git operations against the same clone
notify: sent to Discord: gardener tend: FAILED — Stephenson-Software/SimpleAccountRegistry

The state db shows the false error alongside a genuine success on the same repo five minutes later, from the run that actually held the lock:

2026-09-05T14:05:03+00:00  Stephenson-Software/SimpleAccountRegistry  tend  error  "... is already being worked on by another gardener process ..."
2026-09-05T14:10:38+00:00  Stephenson-Software/SimpleAccountRegistry  tend  tend   "3 issue(s) filed/closed, PR #15 opened, docs sweep ..."

Nothing was wrong with that repo; it was tended successfully minutes later. The alert was noise.

Where it comes from

Both lock handlers construct a run with outcome="error":

  • gardener/cli.pycmd_align's except repo_lock.RepoLockedError (~L559-567)
  • gardener/cli.py_run_tend_dispatch's except repo_lock.RepoLockedError (~L805-811)

_notify_run then maps that unconditionally:

if run.outcome == "error":
    level = notify.Level.ERROR
    title = f"gardener {run.mode}: FAILED — {run.repo}"

For the cycle-accounting half, cmd_overnight's persist_cursor (~L1218) filters only on blocked:

newly_attempted = [outcome.repo for outcome in outcomes if not outcome.blocked]

blocked is set from is_device_global_failure (dispatch.py L422), which matches auth/usage/GitHub-reachability failures — a lock message matches none of them, so blocked stays False and the repo is persisted as attempted. Note that TendResult already carries dispatched=False for this path (L811); that signal is available but is not consulted here.

Suggested direction

A distinct non-error outcome for "skipped, not attempted" would address both halves at once, rather than special-casing the notification: _notify_run could leave it un-alerted (or send it at Level.INFO), and persist_cursor could exclude it from newly_attempted so the repo is retried rather than deferred a whole cycle. Deciding whether such a run should be recorded in the state db at all — it is arguably history worth keeping, just not as an error — is left to the maintainer.

Worth checking as part of any fix: whether the same treatment should apply to the other dispatched=False path (a failed create-dev-loop bootstrap, L764), which is a genuine failure and probably should keep alerting.

Environment

  • gardener at 03a48cd61, --concurrency 4, --strategy random
  • No existing issue covers this; the search was run over all open and closed issues.

drafted by Claude on behalf of Daniel Stephenson

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions