Skip to content

ci: fence fat_driver case-table annotations; section git fetch reads up to 3 times (P6) #4936

Closed as not planned

Description

@noahgift

Re-scoped 2026-10-08 (cop executor): two items. Guard-step timeouts are #4921, not this ticket.

A. Planted fat_driver self-test rows raise live ::error:: annotations

x86-main and every workspace-test shard carried section a: failure, section a: cancelled and external job 'workspace-test': not completed within 12000s as error annotations about a minute in. No section had failed. The lines came from fat_driver.py self-test, whose planted rows print real workflow commands (the table passed 60/60). x86-main runs no external-job wait at all, so its 12000 s line could only come from the table. When the job went red later on another section, the annotation was read as the cause.

Fix: the self-test runs inside GitHub's ::stop-commands::<token> fence, through a wrapper (scripts/ci/fence_workflow_commands.sh); fat_driver.py is unchanged. Falsifier: a real self-test failure still turns the step red, with one live ::error:: naming it. A wiring check is RED on an unfenced self-test and on a fenced real run/wait, whose deadline ::error:: must stay live.

B. Section git fetch is one read; a stalled read is a red section

Sections died on RPC failed; curl 92 HTTP/2 stream N was not closed cleanly: CANCEL (err 8) or curl 56 Recv failure: Connection reset by peer, on every runner host alike (the x86-main and shard reds of PRs 4931-4935 and 4923). One tier-step fetch of main at depth 1 ran 376 s and died, although main's objects were already local. The read did not answer; it was not too big. The egress path is with infra.

Change (operator C343 #9, P6): scripts/ci/fetch_p6.sh replaces all 11 git fetch call sites in ci/sections.yml and the merge_group deepen in scripts/lib/resolve_base.sh. This reduces section-fetch drops and does not clear them. actions/checkout already reads 3 times and still failed on #4927. Those drops, and the stalls behind every drop, need infra's network act.

  • A read that did not answer is read at most 3 times. An answer (missing ref, not found, 401/403/404) is never re-read.
  • Each re-read is one status line.
  • Every read is cut when it runs below 1000 B/s for 60 s.
  • 3 reads with no answer still fail the step.
  • A wiring check is RED on any bare git fetch left in ci/sections.yml.

Checklist

  • A: fence the case tables (2e082ae)
  • B: P6 fetch helper at every section fetch (d7785ea, 2481ab6); reduces drops, the network act is infra's
  • Red/green case tables and mutants in the PR body; the fence confirmed in a real run's log

Activity

  1. changed the title [-]ci(x86-main): section a fails 'workspace-test not completed within 12000s' after 66s — false stop[/-] [+]ci: fence fat_driver case-table annotations; section git fetch reads up to 3 times (P6)[/+] on Oct 8, 2026
  2. 16 remaining items

  3. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    Folded into #5306 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.

    Agent: aprender-clerk

  4. noahgift commented on Oct 14, 2026

    @noahgift
    MemberAuthor

    Row moved: #5306 was split into named masters (TRIAGE-100 master naming), so this ticket's checklist row is now on #5324.

    Agent: aprender-clerk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions