test(transport-ssh): stop fake_ssh racing ETXTBSY against sibling forks

[hartsock]

Summary

agent-mesh-transport-ssh's unix_process tests failed intermittently with
ExecutableFileBusy at the .expect("spawn") sites, blocking the pre-push gate on
any branch — this was found on a docs-only diff that touches no Rust.

fake_ssh no longer writes the file it later execs. The script bytes go to a
staging path that is never executed, and a child process (/bin/cp) creates the
exec target.

Production diff is empty. Production was never exposed: the only production
path is SshTransport::bind → OpenSshClient::system() (transport.rs:164) →
/usr/bin/ssh, and set_mode(0o7xx) appears exactly once in the whole workspace —
inside mod tests > mod unix_process.

Fixes #86

Root cause

Linux tracks write access as i_writecount per open-file-description, not per
descriptor. libtest runs the four #[tokio::test]s as threads in one process
sharing one descriptor table, and glibc's posix_spawn issues clone3 without
CLONE_FILES — so a sibling test thread's spawn duplicates the whole table.

While thread A is inside fs::write(&executable, ..), that duplicate keeps A's
write-side struct file alive past A's own close(): f_count 2→1, so __fput()
never runs, so put_write_access() never runs, so i_writecount stays 1 — and A's
subsequent execve of that same inode is refused by deny_write_access().
O_CLOEXEC is irrelevant: it fires at the forked child's exec, not at fork.

The interference is not cross-inode. Each test has its own tempdir; the write
and the exec are both on A's own fake-ssh. Only the fork is cross-test, and
the forking thread's target program is irrelevant.

Delegating the write removes the precondition rather than retrying around it, and
the ordering is a kernel guarantee rather than a probability: status() returns
only once the copier has exited, and do_exit() runs exit_files() then
exit_task_work() — which flushes the deferred __fput — before exit_notify()
releases our wait4. No descriptor for the executable ever exists in this process
to be inherited.

Rejected alternatives

Option	Why not
Retry on ETXTBSY at the production spawn site	Masks a true alarm — ETXTBSY there means someone holds a write fd on that inode right now, in a module whose docs stress the child is an unauthenticated carrier. Only reduces (a child pinning the fd past the retry budget defeats it — measured: 600 ms hold → still refused after 10 attempts). Also stalls the transport-wide outbound_connect mutex held across connect().
sync_all() before spawn	Measured 54–176× wider window — it holds the descriptor open across the fsync, i.e. exactly the window the bug lives in. (newt-agent added this; its stated rationale is false — fs::write already closes before returning.)
rename() into place	No effect — the leaked duplicate is on the inode; rename only rewrites a directory entry. Measured 54/1600 vs 62/1600 naive.
flock handshake	Introduces this crate's first unsafe (or pushes MSRV 1.75→1.89 via File::lock), and a blocking LOCK_EX on an fd whose duplicate leaked into a long-lived child turns a retryable error into a silent hang.
static Mutex / --test-threads=1	Hides it. The Mutex measurably still fails once any unserialized fork site exists.

Test plan

Measured on gnuc, ext4 /tmp, rustc 1.97.1, 16 cores. The failure rate is
filesystem-dependent — on tmpfs (/dev/shm) the pristine code did not reproduce
at all (write window 5 µs vs 26 µs), so a green run on a memory-backed /tmp proves
less than it looks.

Condition	Result
Pristine, unix_process, ×20	12/20 runs FAILED, all ExecutableFileBusy
Pristine, unix_process, --test-threads=1	4/4 pass
New regression test vs old helper, ×5	5/5 FAILED — 14, 12, 16, 17, 10 of 160 refused
New regression test vs new helper, ×10	0/10 failed
Fixed, unix_process (5 tests), ×30	0/30 failed
Fixed, full crate lib (42 tests), ×5	0/5 failed, 42 passed each
cargo fmt --all -- --check	clean
cargo clippy --all-targets -- -D warnings	clean
.githooks/pre-push (full just check + cov-ci)	OK — the same gate failed 3/3 before

On the regression test

fake_clients_are_executable_under_concurrent_fork_pressure is probabilistic by
necessity, and its doc comment says so. The race window is the open/close
pair inside fs::write, so no safe-Rust test can schedule a sibling fork inside
it on demand. What it does is drive the window to be hit with overwhelming
probability — 4 writer threads × 40 scripts against 4 forker threads churning
fork+exec — and assert not one spawn is refused. P(clean run before the fix)
≈ 3 × 10⁻⁷ on this box. It is not a proxy: it exercises the exact defect through
the exact helper, and fails hard if fake_ssh regresses to an in-process write.

It asserts on ETXTBSY only — a saturated machine may legitimately refuse a fork
with EAGAIN, and failing on that would trade one flake for another.

Cost: 0.06 s, self-terminating.

Residual risk

• Rate is machine- and filesystem-dependent (see above); CI's runner may back /tmp
  with a memory-backed emptyDir.
• Could not be observed on macOS (no macOS box available); posix_spawn there closes
  CLOEXEC descriptors in-kernel at spawn. The change is inert-but-harmless there.
• New dependency on /bin/cp in the test environment — coreutils, and the suite
  already hard-codes /bin/kill and /bin/sleep. Fails loudly, not mysteriously.
• Does not fix the same idiom in sibling repos. test(transport-ssh): fake_ssh write-then-exec races ETXTBSY under parallel tests #86 lists them; the notable one is
  agent-bridle agent-bridle-core/src/sandbox.rs:3440, where the test asserts an exec
  is denied, so an ETXTBSY masquerades as a pass and silently degrades a
  security proof.

Prior art

newt-agent hit the same lineage (#288 → PR #290 → PR #293) and took a retry
approach. This PR takes the structural fix instead, for the reasons in the table above.

🤖 Generated with Claude Code