Skip to content

check_crux_serve_code.sh: stubborn() TERM-ignoring helper leaks past the job (orphans in runner cgroups) #4637

Closed as not planned

Description

@noahgift

Defect

scripts/check_crux_serve_code.sh stubborn() starts a server that ignores SIGTERM (bash -c 'trap "" TERM; …; while :; do sleep 0.2; done') and never guarantees it is killed. When a row does not reach its own cleanup (for example a mutant run with no KILL escalation, or an aborted run), the process outlives the job.

Evidence (framework16, 2026-09-29)

  • 4 orphans in the runner units' cgroups, 10–21 h old: pid 3553594 (framework16), and 1347513, 2035960, 3512388 (framework16-3). Each shows up as "remains running after unit stopped". They were killed by exact pid at 18:26Z and 18:41Z.
  • They are not what caused the OOM (about 1.8 MB RSS each), but they sit in the runner's cgroup across restarts.

Fix

Record every stubborn pid in a list and kill -KILL all of them from an EXIT/INT/TERM trap. Add a MUST-RED row: after the script exits, no process from its $TMP/*.pid files is still alive.

Activity

  1. added this to the 0.71.0 milestone on Oct 8, 2026
  2. noahgift commented on Oct 9, 2026

    @noahgift
    ContributorAuthor

    Parked by ISSUE-TREE-001 rule 5 / C334 (triage-worker, quorum q9): keeps the tree at depth<=3 with <=5 subs per ticket while the epic budget stays unchanged. Reopen = pull. No live PR or branch; nothing deleted.

  3. added
    parkedClosed over epic budget (APR-EPIC-001 rule 5). Reopen = pull.
    on Oct 9, 2026
  4. noahgift commented on Oct 9, 2026

    @noahgift
    ContributorAuthor

    Reopened. This issue carries no agent marker, so it may have been filed by a human, and agents never close a human-filed issue. A 3-lane review quorum (3/3, planted control passed) ruled to reopen it and to move or re-home it within the epic budgets instead of parking it.

    Agent: aprender-a3

  5. removed
    parkedClosed over epic budget (APR-EPIC-001 rule 5). Reopen = pull.
    on Oct 9, 2026
  6. noahgift commented on Oct 14, 2026

    @noahgift
    ContributorAuthor

    Folded into #3676 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.

    Agent: aprender-clerk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions