fix(daemon): stale push locks left servers waiting for game files forever [#58]

[K4ryuu]

Fixes #58, where two servers sat on "Waiting for game files" for days while --doctor reported the host as healthy.

Root cause

Both servers had a push lock directory left over from July with no owner pid in it. Every worker that touched those containers waited on that lock, so their status never left queued. Because the status refresher keeps the timestamp fresh for as long as a worker lives, the egg read that as "the daemon is working" and waited forever instead of falling back to SteamCMD, and the doctor's stuck-state check, which measured the same timestamp, never fired either.

The lock itself was never cleaned because the doctor scanned /var/lock with find, and on Debian/Ubuntu that path is a symlink to /run/lock which find does not descend into without -H. That entire cleanup had been scanning nothing.

What changed

Locks

• _acquire_push_lock is now shared by the cron push and the daemon workers, and takes over a lock whose owner is dead or carries no pid instead of waiting out the full budget
• workers trap TERM/INT/HUP so a service restart runs their cleanup, rather than leaking the lock directory (bash skips the EXIT trap on an uncaught signal)
• the doctor's orphan-lock scan uses find -H, so it actually sees the locks

No more unbounded waits

• the wait for a free worker pool slot is capped at 1h and reports failed, which drops the server to SteamCMD instead of leaving it in limbo
• the doctor measures time spent in a state using the worker's enqueue stamp, not the refreshed timestamp

One worker per server, one daemon per host

• a duplicate worker is skipped while a live one is registered for that container, and a worker only drops the registry entry while it still names its own pid, so siblings no longer unregister each other mid-push
• daemon mode takes a host-wide flock. A hand-started second instance used to wipe the running daemon's worker registry
• starting --daemon next to a running daemon now reports on it (pid, uptime, version, pushes in flight) instead of failing, and new --daemon restart drains in-flight pushes before cycling the service

Cleanup and reporting

• VPK links whose source a CS2 update removed are pruned on both the push and the verify path; the egg skips its own broken-link cleanup while the daemon is authoritative, so they used to pile up
• the doctor no longer counts VPK links belonging to another mount, which resolve only inside the container and were reported as broken on healthy servers
• --test and --doctor bypass the GitHub CDN cache so they always see the branch as it is now; self-update keeps the plain URL since it runs from cron on every host

Testing

misc/protocol-test.sh grew five daemon-side cases: push lock steal/wait, stale link prune, duplicate worker guard, daemon instance lock, and the symlinked lock directory scan. The suite runs green on macOS and in a Debian container, as root and as a normal user. Two of the new cases were verified to fail against the old code.

Confirmed on the reporter's production host: both stuck servers picked up the current build and booted normally, and a separate host was used to verify --daemon reporting and --daemon restart.

Also

The README roadmap gained two entries that came out of this: an update soak window for the automatic self-update, and signed releases as the longer-term supply-chain answer.