Headless agents in GitHub Actions can stall silently: background tasks, green checks, and zero output

[cmungall]

Summary

Agents run headlessly in GitHub Actions (claude -p via claude-code-base-action) can end their turn waiting for an event that will never arrive, exit 0, and produce no visible output at all. The job shows a green check. Nothing is committed, no PR is opened, no comment is posted on the triggering issue.

We should document this failure mode and harden our workflow templates against it. It is not specific to one repo or one agent — it is a structural property of running an interactive-assumption agent in a non-interactive harness.

Worked example

go-ontology run 31436984094, responding to a mention on geneontology/go-ontology#29437.

The agent launched ontology pre-validation as a background Bash task:

Bash {
  "command": "cd .../src/ontology && make travis_build 2>&1 | tail -80",
  "timeout": 600000,
  "run_in_background": true
}
// → "Command running in background with ID: bze8lu1so.
//    You will be notified when it completes."

It then made all three requested ontology edits correctly, polled the output file twice, saw it still empty, and ended its turn with:

"I'll wait for the pre-validation completion notification."

That was the final message. From the run artifact (claude-execution-output.json):

{ "is_error": false, "subtype": "success", "stop_reason": "end_turn",
  "terminal_reason": "completed", "duration_ms": 163587 }

Outcome: exit 0, green check, and

• nothing posted to #29437 beyond the workflow's own "🤖 Working on it..." comment
• branch ai4c-agent-issue-29437-run10337 created locally, never pushed
• edits left in terms/*.obo, never even checked back into go-edit.obo
• 2m44s used against a 30-minute budget

A near-miss worth noting: in its last two actions the agent ran ToolSearch("select:Monitor"), loading the schema for Monitor — the tool that does block inside a turn and would have worked — and then ended the turn without calling it.

(A later re-run, 31441317508, completed the same task and opened PR #32434. So this is intermittent, which makes it more insidious, not less.)

Root cause

The "you will be notified when it completes" contract requires a subsequent turn to deliver the notification. In -p headless mode there is no interactive user, so no subsequent turn is ever triggered. end_turn terminates the process.

The agent is not wrong about how background tasks work interactively. It simply has no way to know it is in a harness where that contract cannot be honoured.

Two compounding factors:

1. The failure is silent. Exit code 0, is_error: false, subtype: "success". Nothing in the Actions UI distinguishes it from a successful run.
2. Our template has no result-posting step. The ai-agent.yml prompt instructs the agent to communicate via gh itself, so all user-visible output is downstream of the stall.

Mitigations

Verified locally against Claude Code unless noted.

1. Remove the trigger: CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1

Setting this env var removes run_in_background from the Bash tool schema entirely. Tested:

InputValidationError: Bash failed due to the following issue:
An unexpected parameter `run_in_background` was provided

It also covers the auto-backgrounding path (Claude Code will otherwise move a long-running foreground command to the background on its own, via CLAUDE_CODE_AUTO_BACKGROUND_TIMEOUT_MS). Verified: with the flag set, a 150s foreground command — comfortably past the 120s default Bash timeout — ran to completion in-turn, session duration 161.8s. Not tested at multi-minute scale beyond that.

- name: Run Claude Code
  uses: anthropics/claude-code-base-action@main
  env:
    CLAUDE_CODE_DISABLE_BACKGROUND_TASKS: "1"

⚠️ This env var is recognized by the CLI but is not in the public docs. Treat it as working-but-unsupported and keep #2 as the real safety net.

2. Stop hook — the general fix

A Stop hook fires when the agent tries to end its turn; returning {"decision": "block", "reason": "..."} forces it to continue. stop_hook_active in the hook input guards against infinite loops.

Our completion criterion is objective — did it post a comment? — so the hook can check reality rather than nag:

#!/usr/bin/env bash
# .github/hooks/require-comment.sh
input=$(cat)
[ "$(jq -r '.stop_hook_active' <<<"$input")" = "true" ] && exit 0  # already forced once

posted=$(gh issue view "$ISSUE_NUMBER" --repo "$GITHUB_REPOSITORY" --json comments \
  --jq '[.comments[] | select(.author.login=="ai4c-agent[bot]" and (.body|test("Working on it")|not))] | length')

if [ "$posted" -eq 0 ]; then
  jq -n '{decision:"block", reason:"You have not posted a result comment. This is a non-interactive GitHub Actions run: ending your turn terminates the session and the requester sees nothing. Finish now — commit, push, open the PR if applicable, and post via `gh issue comment`. Do not wait for background task notifications; they are never delivered in this environment."}'
fi
exit 0

Both claude-code-action and claude-code-base-action accept a settings input (JSON string or path to a settings file), which is how hooks get injected:

settings: |
  {"hooks": {"Stop": [{"hooks": [{"type": "command",
    "command": "${{ github.workspace }}/.github/hooks/require-comment.sh"}]}]}}

This catches every premature-stop variant, not just background waiting: "let me know if you'd like me to proceed", giving up after an error, running out of plan.

3. PreToolUse hook, if we prefer not to depend on an undocumented env var

The PreToolUse hook input includes tool_input.run_in_background, so with matcher Bash:

input=$(cat)
if [ "$(jq -r '.tool_input.run_in_background // false' <<<"$input")" = "true" ]; then
  jq -n '{hookSpecificOutput:{hookEventName:"PreToolUse",permissionDecision:"deny",
    permissionDecisionReason:"Background tasks do not work in this non-interactive run — completion notifications are never delivered. Re-run in the foreground with an explicit timeout."}}'
fi
exit 0

4. Workflow-level backstop (if: always())

Hooks do not run if the process itself dies. A final step that reads claude-execution-output.json and posts the result field when no comment landed also catches OOM, job timeout, and API errors. Cheap, and it converts silent-green into a visible "agent stalled, here is its last message".

5. Evaluate moving from base-action to claude-code-action v1

We currently use claude-code-base-action, the raw runner, which does nothing about communication — hence "green run, zero output". v1 manages a tracking comment itself (creates it, updates it with progress and results). base-action is now just a mirror of the base-action subdirectory inside claude-code-action.

Related constraint: the 10-minute Bash ceiling

Worth documenting alongside this, because it is what pushed the agent toward backgrounding in the first place: the Bash tool's max timeout is 600000ms (10 minutes) — a hard cap, not a raisable default. The agent in the failing run passed exactly 600000.

So make travis_build cannot be a single foreground Bash call. Disabling background tasks converts the silent stall into an honest 10-minute kill; it does not make a full ODK build fit. Options:

1. Targeted pre-check — robot reason / robot verify on the changed terms, which completes in minutes. Best fit when the agent is editing three stanzas, not rebuilding the ontology.
2. Move validation out of the agent — let it commit and open the PR, and let the existing qc.yml PR checks do the real validation. This is what they are for, and what the successful re-run effectively did.

Duplicating CI inside a 30-minute agent budget bounded by a 1-hour app installation token is the wrong division of labour regardless of this bug.

Proposed actions

• Add a best-practices doc: "Running agents headlessly in CI", covering the no-interactive-turn contract, silent success, and the mitigation layers above
• Add CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1 to the workflow templates
• Add a reusable Stop hook + if: always() backstop to the templates
• Document the 10-minute Bash ceiling and the "let PR checks validate" pattern
• Add prompt guidance: never wait on background notifications in CI; prefer PR checks over in-agent full builds

Related: #58 (actions template / ODK image)

[cmungall]

[human]

re:

Move validation out of the agent — let it commit and open the PR, and let the existing qc.yml PR checks do the real validation. This is what they are for, and what the successful re-run effectively did.

No. bad idea. Creates unnecesary noise for curators. Also what happens if it fails e.g on a trivial syntax errror? how does the agent pick up and carry on

[cmungall]

Retracting mitigation option 2 ("move validation out of the agent, let qc.yml PR checks validate") from the 10-minute Bash ceiling section above. The objection is correct and the option should not be adopted.

Two reasons it fails, the second being the deeper one:

1. A red PR is worse than no PR. It moves work onto a curator rather than off them. Noise in the review queue is the cost the agent was supposed to eliminate.
2. There is no pickup mechanism. The agent process has exited by the time a PR check reports. A stateless agent cannot "see the failure and carry on" — a trivial syntax error would sit there until a human noticed it.

What I got wrong

I asserted the 10-minute ceiling as if it constrained the whole workflow. It does not: 600000ms is a cap on the agent's Bash tool, not on GitHub Actions. A workflow step has no such cap — it is bounded only by the job's timeout-minutes.

So the correct move is not to push validation downstream, and not to make the agent wait better. It is to move the expensive build out of the agent's tool call into a workflow step, and re-invoke the agent between steps.

Corrected architecture

- name: Agent — edit          # edits + local commit. No push, no PR.
- name: Validate              # make travis_build — no 10-min cap applies here
  id: v1
  continue-on-error: true
- name: Agent — repair        # only if v1 failed; prompt carries the validation log
  if: steps.v1.outcome == 'failure'
- name: Re-validate
  id: v2
  if: steps.v1.outcome == 'failure'
  continue-on-error: true
- name: Agent — publish       # push + PR + comment, only when green
  if: steps.v1.outcome == 'success' || steps.v2.outcome == 'success'
- name: Report failure        # still red → explanatory comment on the issue, still no PR

Re-invocation with the failure log is the pickup mechanism — the only one available to a process that cannot sit and wait. The loop belongs in the workflow, not in the turn.

This satisfies both objections directly: no PR is ever opened unless validation is green, so curator noise is zero; and a persistent failure produces an explanatory comment rather than a broken PR.

Bounded cost: two or three agent invocations, not an open loop. The tradeoff is context loss between invocations, mitigated by passing the diff and the validation log into the repair prompt.

Tiering, with measured numbers

So that it doesn't take 20 minutes to discover a typo, the in-agent tier should do fast rejection while the agent still holds full context:

Check	Local timing	Fits the 10-min tool cap?
robot convert -i go-edit.obo -f ofn (parse)	1m27s	yes, comfortably
+ robot reason -r ELK -i go-edit.obo	~4m30s	marginal on a 4-core runner
full make travis_build	≫ that	no

Measured on go-edit.obo, 48,355 [Term] stanzas, on a local Mac. The gap at the bottom row is because travis_build is far more than reasoning: it depends on go-edit-filtered-xrefs.ofn, then merges four imports, runs three SPARQL update queries, relax/reduce/annotate, and two verify passes over the QC query set. Building imports alone can dominate.

Two caveats against over-reading these numbers:

• GitHub standard runners are 4-core/16GB and will be slower than the machine above; ELK over GO may also press on ROBOT's default heap.
• Reasoning go-edit.obo in isolation is a screen, not a substitute. It merges no imports and no taxon constraints, so it misses violations that only surface post-merge — a good share of what travis_build's verify queries exist to catch.

That second caveat is the load-bearing one: the in-agent tier must not try to be authoritative, or we are back to needing 20 minutes inside a single turn. Fast rejection in-agent; authoritative build in a step.

Constraints to design around

• The app installation token is valid 1h. travis_build plus two or three agent invocations will not fit the current timeout-minutes: 30, so that needs raising and the token re-minting before the push step.

Amended checklist

• Document the "let PR checks validate" pattern — retracted, see above
• Document the validate-in-step + re-invoke-agent loop as the standard template shape
• Document the tiering rule: in-agent checks are for fast rejection only, never authoritative
• Document the 10-minute Bash tool cap explicitly as a tool limit, and note that steps are not subject to it

[cmungall]

I went back and measured instead of reasoning from the single failing run. Withdrawing the validate-in-step architecture from my previous comment — its central premise is false.

Method

Downloaded all 121 non-expired claude-response-* artifacts from ai-agent.yml runs in geneontology/go-ontology (2026-05-13 → 2026-08-11), parsed every Bash tool call out of the execution transcripts, and derived wall-clock per call from the timestamp delta between each tool_use and its tool_result.

What the agent actually does, unprompted

Normalised position of each command class within its run (0.0 = first bash call, 1.0 = last):

stage	median pos	calls	median wall-clock
obo-grep (inspect)	0.33	653	0.1s
obo-checkout	0.42	86	<1s
obo-checkin	0.48	75	0.6s
robot convert	0.54	49	24s
robot verify	0.66	38	120s
robot reason	0.70	47	63s
make travis_build	0.83	153	444s
git commit	0.87	40	—
git push	0.93	57	—
gh issue comment	1.00	54	—

Cost rises monotonically with position. The agent has converged on a sensible escalating-validation ladder on its own: cheap term inspection in a tight loop, medium-cost structural and reasoning checks as it converges, full build once before committing, then publish.

Notably it does not defer the full build to CI: make travis_build appears in 58/121 runs, first invocation at median position 0.62.

Why the previous comment was wrong

I claimed travis_build was "≫ 10 minutes" and could not be a foreground tool call. Measured:

• 91 foreground travis_build calls; zero exceeded the 600s tool cap
• median 1.2s, because make short-circuits on repeat invocations within a run
• excluding those no-ops, 27 calls actually rebuilt: min 356s, median 444s, p90 458s, max 494s — 18% headroom at worst
• exactly 1 call exceeded 480s

The estimate came from cold local timings on a laptop. The odkfull container is 2–4× faster than that machine (robot convert 24s vs 87s; robot reason -r ELK 63s vs 180s), the reverse of the caveat I attached. The agent's existing in-turn ladder works; restructuring the workflow into edit → validate-step → repair → publish would add cost and context loss to solve a problem that is not occurring.

The genuine timeout hazard is smaller and different: 5 occurrences of Command timed out after 2m 0s — the default Bash timeout biting when the agent omits an explicit one — against 1 occurrence of the 10-minute cap.

Correction: Monitor is not a safe alternative to background bash

My first comment described Monitor as blocking within the turn. It is not — it is notification-based, with the same delivery requirement as run_in_background, and it fails identically in headless mode. Run 28375280482 ended on "Build is materializing. Waiting for monitor."

Stall taxonomy (n=121)

async-wait surface	runs touching it	runs stalled on it
run_in_background bash	14	2
Monitor tool	8	2
Agent subagent	2	1
union	20	5

Plus 3 runs that completed their work and never posted it ("Research complete.") — a prompt-compliance failure rather than a stall, but with an identical outcome for the requester.

Silent runs: 8/121 (6.6%). Comment rate: 93.4%.

Revised recommendation

The ordering in the issue body is wrong. Corrected priority:

1. Stop hook — load-bearing. It is the only mitigation covering all 8 silent runs, because it gates on the observable outcome (was a comment posted?) rather than on any particular route to failing. All three async surfaces and the prompt-compliance failures collapse into one check.
2. CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1 — cheap defence in depth, but it covers only 2 of 5 stalls. Not the highest-value change, as originally claimed.
3. if: always() backstop — unchanged; still the only thing that survives process death.
4. Prompt guidance on explicit timeouts — new, and better supported by the data than anything about the 10-minute cap: the 2-minute default is the more frequent killer.
5. Validate-in-step + re-invoke architecture — withdrawn, premise false.

Amended checklist

• Document the "let PR checks validate" pattern — retracted (curator noise, no pickup mechanism)
• Document the validate-in-step + re-invoke loop — withdrawn, travis_build fits in-turn
• Document the observed escalating-validation ladder as the reference pattern, since the agent already converges on it
• Stop hook in the templates, as the primary mitigation
• Prompt guidance: always pass an explicit timeout (the 2m default is the common failure); never wait on background tasks, Monitor, or subagent notifications in CI
• Note that Monitor and subagents share the background-task failure mode — any "you will be notified" contract is void headlessly

[cmungall]

Amending the Stop hook recommendation. A concern was raised that a hook gating on "did it post a comment?" could trap the agent in a paid loop if it has lost the ability to post at all. I tested it, and the concern is well founded — with a worse result than expected.

Claude Code does not self-bound a blocking Stop hook

Two runs, identical trivial prompt, hook logging every invocation:

hook	times fired	stop_hook_active	outcome
blocks unconditionally	6	false, then true for every subsequent fire	never self-terminated; killed by --max-turns 6, is_error: true
checks stop_hook_active, exits 0 when true	2	false, then true	exactly one forced continuation, then clean stop

The load-bearing finding: stop_hook_active is set true after the first block and never resets, and the runtime does not stop looping on its own. The unguarded hook ran until --max-turns cut it off. Nothing else intervened.

So [ "$active" = "true" ] && exit 0 is not defensive boilerplate — it is the entire bound. Any Stop hook published without it is an unbounded spend loop.

Bug in the hook posted earlier in this issue

posted=$(gh issue view ... --jq '... | length')
if [ "$posted" -eq 0 ]; then ...

If gh fails — expired installation token, revoked permission, API blip — posted is empty and [ "" -eq 0 ] raises "integer expression expected". It happens to fail open only because of the trailing exit 0. That is accidental, not designed, and it is the exact scenario in which you least want to be relying on an accident.

A completion check that cannot run must never be able to trap the agent. Corrected, failing open at every step:

#!/usr/bin/env bash
# Blocks ONCE if the agent stops without reporting. Fails OPEN on any error.
set -uo pipefail
input=$(cat)

# Hard bound. Verified: without this, stop_hook_active stays true and the
# loop runs until --max-turns. Non-negotiable.
[ "$(jq -r '.stop_hook_active // false' <<<"$input")" = "true" ] && exit 0

# Block only on an AFFIRMATIVE "no comment". If the API is unreachable we
# cannot know, so allow the stop.
out=$(gh issue view "$ISSUE_NUMBER" --repo "$GITHUB_REPOSITORY" --json comments 2>/dev/null) || exit 0
posted=$(jq '[.comments[] | select(.author.login=="ai4c-agent[bot]"
          and (.body|test("Working on it")|not))] | length' <<<"$out" 2>/dev/null) || exit 0
[[ "$posted" =~ ^[0-9]+$ ]] || exit 0
[ "$posted" -gt 0 ] && exit 0

jq -n '{decision:"block", reason:"You have not posted a result comment. This is a non-interactive GitHub Actions run: ending your turn terminates the session and the requester sees nothing. Finish now — commit, push, open the PR if applicable, and post via `gh issue comment`. Do not wait for background task, Monitor, or subagent notifications; they are never delivered in this environment."}'
exit 0

Bounding spend regardless of hook correctness

The run distribution measured earlier (n=121) gives concrete limits to set:

• --max-turns in claude_args — a deterministic ceiling, and the only mechanism that actually stopped the runaway in the test above. Observed legitimate turn counts: median 31, p90 62, max 103. --max-turns 150 leaves genuine headroom while capping a loop.
• timeout-minutes — currently 30, the ultimate backstop since it kills the job regardless of agent state. Note that the longest legitimate run observed was 25.8 min, so real work already brushes this limit; raising it buys headroom at the cost of runaway exposure.
• Observed cost: median $1.59/run, p90 $2.92, max $4.58, $198.18 across all 121 runs. A fully runaway loop is wall-clock-bounded by timeout-minutes, so roughly $5–6 — bad, not unbounded.

The tradeoff, stated plainly

The hook trades a measured 6.6% silent-failure rate for a bounded risk of one extra agent iteration (~$1.59 median) in the case where the completion check itself is failing.

For anyone unwilling to accept even that: the if: always() backstop step carries no loop risk at all, because it runs outside the agent process. It catches every silent run after the fact and reports it, rather than preventing it. That is a weaker guarantee but a strictly safer one, and it is a reasonable default for repos where an unattended spend loop is scarier than a missed response.