model_ladder: a CPU serve probe held the GPU lock past its probe and stalled the same ladder's GPU leg (cause open) #4877

What happened (cause open)

During a scripts/model_ladder.sh --only qwen35-0.8b-q4km run on a CUDA host, a CPU-backend apr serve run … --no-gpu, started by apr_locked (flock wrapper → apr), held the fleet GPU lock for about 26 minutes. During that time the same ladder's GPU apr run leg sat in flock -w waiting for that lock. The serve was idle, and its probe's output never reached a receipt. Killing the serve by its recorded pid released the lock, and the GPU leg proceeded.

So a serve probe's lifetime overlapped a later leg of the same ladder. Either teardown did not run on that path, or it ran and printed something other than clean with the server still alive. The run left no teardown field to say which, because the receipt for that rung was not written.

Why it matters

A CPU serve that holds the GPU lock past its probe serializes every GPU user on the box behind an idle process. The ladder's own LOCK_WAIT (default 1800 s) then turns that into a long stall followed by a decline.

Asks for whoever picks this up

• Find which path lets ladder_serve_probe return, or the rung advance, with the serve still alive. Make the receipt record teardown on every exit path, including an early exit.
• Consider whether a --no-gpu serve needs the GPU lock at all, or should take a separate lock.
• Add a case to scripts/check_model_ladder.sh that fails when a serve child survives its probe.

Build: apr 0.70.1 6192aa9b8. A clean rerun did not reproduce it, so this is one occurrence, not a measured rate.