cleanup(brain): retire the legacy supervisor lock after every runnable checkout is bridge-aware

[adrianwedd]

Follow-up to #221 / #223. This is a removal gate, not a cleanup ticket: the work is
the proof, and the deletion is the last step of it. Nothing here should be actioned
because the code "looks like dead migration cruft".

Acceptance invariant

After migration, supervisor authority is keyed solely to the tmux socket namespace.
Checkout location must have zero effect on whether two supervisors contend.

Why the bridge exists (do not re-derive this from the comments)

The bridge exists solely to prevent a mixed-version deployment from creating the exact
two-supervisor window #221 fixed. Removing it early is a regression in authority
containment, not cleanup debt.

#223 re-keyed the guard from brain_root()/.supervisor.lock (checkout-relative) to
<socket>.supervisor.lock (host-global). A pre-bridge binary knows only the first
key. So for as long as one can be started, a post-bridge supervisor must hold both
locks, or the two versions guard different inodes and both start.

The cost, which is the thing that gets removed here: while the bridge holds, two
supervisors on different sockets in the same checkout also contend, because both
must take that checkout's legacy lock. The migration-era namespace is the pair
(socket, checkout).

The gate — every item must be proved before touching _BRIDGE_HOLDS_LEGACY_LOCK

Audit run 2026-08-19, immediately after #223 merged as 663b378d. Two items fail
today. Re-run all of it at removal time; do not trust this table's age.

G1 — Enumerate every checkout on the host that can plausibly start bin/px-brain

find /home/pi -maxdepth 3 -name px-brain -type f

Checkout	HEAD	Branch	Contains 663b378d	legacy_supervisor_lock_path refs
/home/pi/picar-x-hacking	663b378d	master	YES	3
/home/pi/px-brain-wt	efe22d42	e2e/brain-holder-verify	NO	0

❌ FAILS. /home/pi/px-brain-wt is a runnable pre-bridge checkout. Its
brain_daemon.py contains no reference to the socket lock at all, so a supervisor
started from it guards only state/brain/.supervisor.lock under that worktree. This is
precisely the binary the bridge exists to lose to.

Resolution is either: rebase/merge that worktree past 663b378d, or delete it if the
e2e/brain-holder-verify work is finished. Prove it per checkout, not per branch —
a merged branch whose worktree still sits on an old commit is still a runnable
pre-bridge binary.

G2 — Record each checkout's commit and prove it contains the bridge

for d in $(find /home/pi -maxdepth 3 -name px-brain -type f -printf '%h\n' | xargs -n1 dirname); do
  git -C "$d" merge-base --is-ancestor 663b378d HEAD && echo "$d OK" || echo "$d PRE-BRIDGE"
done

Ancestry, not a grep for the function name — a checkout can carry the symbol from a
partial cherry-pick without carrying the dual acquire.

G3 — Inspect the active unit's WorkingDirectory and executable path

systemctl cat px-brain.service | grep -E '^(WorkingDirectory|ExecStart)='

WorkingDirectory=/home/pi/picar-x-hacking
ExecStart=/home/pi/picar-x-hacking/bin/px-brain

✅ PASSES. One unit, pointing at the bridge-aware checkout.

G4 — Confirm no unit, cron, wrapper, shell script or other launcher points at a pre-bridge checkout

ExecStart is not the only way in. Sweep at minimum:

systemctl list-units --all | grep -i brain
grep -rl 'px-brain' /etc/systemd/system /etc/cron* /home/pi/*/bin 2>/dev/null
crontab -l; sudo crontab -l
ssh M5.local 'crontab -l'   # M5 pushes to this host; check it cannot invoke px-brain

Not yet swept — the M5 cron leg in particular has not been checked.

G5 — Confirm the running production supervisor is bridge-aware

Not "the unit points at a bridge-aware checkout" (G3) — the live process. A
bridge-aware supervisor holds two locks; a pre-bridge one holds only the legacy lock.
That is directly observable:

ls -la /tmp/tmux-1000/*.supervisor.lock          # socket lock present?
sudo lsof "$(git rev-parse --show-toplevel)/state/brain/.supervisor.lock"

Observed 2026-08-19:

socket lock:  ABSENT   (/tmp/tmux-1000/ holds only the px-mind socket itself)
legacy lock:  python3  PID 740  pi  3uW  .../state/brain/.supervisor.lock
PID 740 started 12:16:19 AEST = 02:16 UTC; #223 merged 06:18 UTC

❌ FAILS. The running supervisor predates the merge and holds only the legacy lock.
It becomes bridge-aware on systemctl restart px-brain and not before. Until then the
host is in the mixed state the bridge is written for — which is the bridge working, not
a problem to fix by deleting it.

G6 — Only then, remove

In one commit:

• _BRIDGE_HOLDS_LEGACY_LOCK and the if not ... branch in acquire_supervisor_lock()
• legacy_supervisor_lock_path()
• the second acquire, its failure path (the _drop_lock(fd)-then-refuse branch), and
  _legacy_fd throughout, including in release_supervisor_lock()
• TestMigrationBridge in tests/test_supervisor_lock_scope.py
• TestContention::test_after_removal_the_socket_alone_is_the_namespace (case 4 —
  it exists to describe this end state and is redundant once reached)
• the migration paragraphs in CLAUDE.md and acceptance criterion 2 of
  docs/specs/2026-08-19-supervisor-lock-scope-migration-design.md

Post-removal test expectations

TestContention is written so that removal shows up as a specific, single flip —
if anything else moves, the removal broke containment:

Case	Test	Before	After
1	test_two_checkouts_on_one_socket_cannot_both_acquire	contend	contend (unchanged — this is #221's fix)
2	test_different_sockets_and_different_state_roots_coexist	coexist	coexist (unchanged)
3	test_different_sockets_in_one_checkout_contend_during_migration	contend	flips to coexist — rewrite the assertion and drop "during migration" from the name
4	test_after_removal_the_socket_alone_is_the_namespace	passes under simulation	deleted

Case 1 staying red-for-a-second-supervisor is the invariant. A removal that also
relaxes case 1 has re-opened #221.

Notes for whoever picks this up

• _BRIDGE_HOLDS_LEGACY_LOCK is deliberately not env-readable. Do not add an
  override to "test removal in production" — flipping it live is exactly the two-
  supervisor window. Case 4 simulates the end state in-process; that is the supported
  way to see it.
• Re-run the whole audit at removal time. G1 and G5 are both time-sensitive: a new
  worktree or an un-restarted daemon re-opens them silently.
• Per CLAUDE.md, do not run the full suite on SPARK — CI is the gate.

Refs #221, #223.

[adrianwedd]

Gate audit update — G1 ✅ PASS, G5 ✅ PASS. G4 still open, so removal remains blocked.

G1 — neutralised the pre-bridge worktree

/home/pi/px-brain-wt was assessed before removal and was disposable on every axis:

worktree status --porcelain : clean (no uncommitted work)
git log master..HEAD        : empty (no unique commits)
git branch --merged master  : e2e/brain-holder-verify IS merged
process cwd in that tree    : none
.supervisor.lock inside it  : none

Removed with git worktree remove. The branch and its commits are untouched
(e2e/brain-holder-verify still resolves to efe22d42) — only the checkout directory
is gone, so this is reversible with git worktree add.

Re-audit:

$ find /home/pi -maxdepth 3 -name px-brain -type f
/home/pi/picar-x-hacking/bin/px-brain

One runnable px-brain on the host, in the bridge-aware checkout at 663b378d.

Also worth recording for future audits: five worktrees remain (px-blogqa-wt,
px-master-baseline, px-upstream-wt, px-vision-wt, plus the primary). The four
non-primary ones sit at commits that predate bin/px-brain existing and carry no such
file — checked individually, not inferred. That is a property of their current commits,
not a permanent one: a rebase or checkout in any of them re-opens G1 silently. Re-run
the find at removal time.

G5 — production supervisor is now bridge-aware

Pre-restart state confirmed first: no current.json in either session (nothing in
flight), empty inboxes, unit → /home/pi/picar-x-hacking which contains 663b378d,
incumbent PID 740 holding only the legacy lock.

sudo systemctl restart px-brain — no tmux session was restarted by hand.

Proof	Result
exactly one supervisor	✅ MainPID 76299; sole holder of both locks
PID changed from 740	✅ 740 → 76299
socket lock exists, held by that PID	✅ python3 76299 pi 3uW /tmp/tmux-1000/px-mind.supervisor.lock
legacy lock held by the same PID	✅ python3 76299 pi 4uW .../state/brain/.supervisor.lock
resident tmux sessions survived	✅ spark-brain + spark-io both present
readiness reaches healthy	✅ both validated on claude-haiku-4-5-20251001
no duplicate-supervisor / start storm	✅ exactly one start record, pid=76299 instance=465ac1f45b63

Singleton-ness is proved by the locks rather than by process counting, which is the
stronger statement: a second supervisor could not hold the socket lock, so "one holder"
is "one supervisor". (A first pass using pgrep -f 'bin/px-brain' reported 2 — it was
matching the auditing shell's own command line. Noted so the next person doesn't repeat
it.)

The journal's seven Found left-over process ... Ignoring lines are KillMode=process
behaving as designed: the supervisor restarts, the sessions it supervises do not die.

G5 PASS.

Three observations from the restart — none blocking, none caused by #223

1. A supervisor restart triggers an immediate recycle ... reason=nightly of both
   sessions, at 16:43 local rather than the 02:00 Hobart window. Both sessions lost
   their context and re-handshaked (spark-brain at attempt 2, spark-io after one
   handshake_failed and a recreate, then validated). Recovery was normal and
   unattended, but a restart is evidently not context-preserving, and the nightly
   label is wrong when it fires this way. Pre-existing; worth its own issue.
2. Holder tmux clients accumulate across restarts. Four attached clients now exist
   for two sessions — the pre-restart pair survived KillMode=process and the new
   supervisor created its own. Harmless for send-keys (which needs ≥1 attached), but
   it leaks one client pair per restart. Pre-existing; worth its own issue.
3. The two supervisor_already_running records at 06:05:20Z are mine, not an
   incident. They came from a diagnostic script I ran while reviewing fix(brain): scope the supervisor guard to the socket it guards #223 (parent
   66465, children 66471 socket-scoped / 66472 legacy-scoped) which set
   PX_BRAIN_TMUX_SOCKET and PX_STATE_DIR but not LOG_DIR, so its refusals landed in
   the production log. Left in place rather than edited out — but flagged here because
   this is exactly the fix(tests): in-process tests write to production logs and bypass the supervisor guard — falsified brain-daemon forensics #221 pollution class, and the only reason it was attributable in
   thirty seconds instead of being re-litigated as a concurrency event is that fix(brain): scope the supervisor guard to the socket it guards #223 item 3
   put pid/instance/boot_id on every record. The lesson generalises past pytest:
   any ad-hoc script that imports pxh inherits production log paths unless it sets
   LOG_DIR.

Still blocking removal

G4 — no unit, cron, wrapper, shell script or other launcher points at a pre-bridge
checkout — has not been swept. The M5 cron leg in particular is unchecked. Until G4 is
proved, do not touch _BRIDGE_HOLDS_LEGACY_LOCK.

Deployment rule this established, beyond this migration

Configured code is not running code. Verify live mechanism and state, not merely the
unit that would launch on the next restart.

G3 passed all along — the unit pointed at the bridge-aware checkout — while G5 failed,
because the daemon had been up since before the merge. The dual-lock bridge doubles as a
precise version probe: one lock = pre-bridge, two locks = bridge-aware. That is
better evidence than reading WorkingDirectory and assuming the daemon got the memo.

[adrianwedd]

Gate re-audit 2026-08-26 — G1 has re-opened. Removal remains blocked.

Per this issue's own instruction ("Re-run the whole audit at removal time. G1 and
G5 are both time-sensitive: a new worktree or an un-restarted daemon re-opens
them silently") — it did exactly that.

$ find /home/pi -maxdepth 3 -name px-brain -type f
/home/pi/px-yield-alive-205/bin/px-brain
/home/pi/picar-x-hacking/bin/px-brain

G1 ❌ FAILS. /home/pi/px-yield-alive-205 is a second runnable px-brain
checkout that did not exist at the 2026-08-19 audit, when the tree was reduced to
one. Whether it is pre-bridge is the G2 question and has not been checked; either
way G1's own criterion ("enumerate every checkout that can plausibly start
bin/px-brain" → exactly one) is not met.

This is the second time a worktree created for unrelated work has silently
re-opened the gate. Worth noting as a property of the gate rather than a
complaint about the worktree: G1 is not a state that can be reached and held,
only sampled — which is an argument for automating it (a preflight check in
bin/px-brain itself, or a test that greps the host) rather than re-running it
by hand at some future removal time.

G3 ✅ still passes — one unit, WorkingDirectory=/home/pi/picar-x-hacking,
which is deployed at 5b9070de (well past 663b378d).

G4 — still not swept. No px-brain reference in pi's crontab
(crontab -l | grep -i brain → none). The M5 cron leg remains unchecked, as it
has since this issue was filed.

G5 — not re-verified this pass (the supervisor has restarted many times
since; verify at removal time, not now).

_BRIDGE_HOLDS_LEGACY_LOCK = True is unchanged at src/pxh/brain_daemon.py:1015
and must stay that way.