Skip to content

model_ladder: GPU ownership check reads the ladder's own apr as FOREIGN when $APR is under a directory symlink (CONTENDED, exit 75) #4876

Closed as not planned

Description

@noahgift

What happened

scripts/model_ladder.sh apr_exclusive() (line 107) sets GPU_OWNED_PREFIX="$(readlink -f "$APR")", which is the symlink-resolved path. The GPU driver's process listing reports the path the binary was invoked through. When $APR sits under a directory symlink (a scratch mount linked to another volume), the two strings differ. scripts/lib/gpu_exclusive_run.sh then classifies the ladder's own apr process as foreign:

decline: ENV the GPU was not exclusive for apr run qwen35-0.8b-q4km (cuda) -- gpu_exclusive_run: CONTENDED -- foreign GPU process(es) during the run, result refused:     <pid>, <symlinked-dir>/apr ...

The pid in that line is the leg's own apr run. Every GPU leg on such a host declines CONTENDED (exit 75), so no GPU verdict can be produced there. It fails closed, never as a false pass. Rerunning with $APR under its real path gives normal verdicts (same build: apr 0.70.1 6192aa9b8).

Repro

  1. ln -s /real/dir /link, then put the apr build under /real/dir.
  2. APR=/link/apr bash scripts/model_ladder.sh --only <rung> on a CUDA host.
  3. The GPU leg declines CONTENDED, naming its own pid.

Fix direction (not done here)

Compare like with like. Either resolve each reported process path the same way (readlink -f /proc/<pid>/exe, or the reported path through readlink -f) before the prefix test, or match ownership by pid ancestry. Add both forms to the guard's case table in scripts/check_model_ladder.sh: a symlinked invocation must read as owned, and a different binary must still read as foreign.

Activity

  1. added this to the backlog milestone on Oct 8, 2026
  2. noahgift commented on Oct 14, 2026

    @noahgift
    ContributorAuthor

    Folded into #4701 as a checklist row (TRIAGE-100, 2026-10-14): the open-ticket cap is 100, so this ticket is now tracked on that master and closed as not planned. Nothing is lost: reopen it when its work resumes, and an open PR for it reopens it.

    Agent: aprender-clerk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions