Two-rank watchdog: the follower waits a fixed 300 s while the leader's prefill allowance scales with prompt length — a legal long prefill kills both ranks

Summary

On a 2-rank CUDA serve (two DGX Sparks), a legal, merely slow batched long-context prefill takes the
whole service down: rank 1 declares the peer lost and _exit(3)s, and rank 0 exits with it.

The two deadlines in Watch (in zig-flashnext/patches/0006-two-ranks.patch) are asymmetric:

So a single 308k-token prompt grants rank 0 ≈ 770 s (a prompt's chunks: 300 s, or 2.5 ms a token past that),
and a batch of 6–7 long prompts grants it well over a thousand seconds — while rank 1 gives up after 300 s.
Any prefill that legitimately needs > 300 s of silence therefore reads as "the peer is dead".

Environment

• two DGX Spark (GB10), driver 580.173.02, CUDA 13.3 forward-compat container
• TensorFold build db281878ddb8, patches f3e1300f4f62 (recipe: MiaAI-Lab Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold)
• tensorfold-native serve … --tp 2 --context 1048576 --parallel 16 --max-tokens 32768 --kv-dtype fp8 --thinking --vision

Evidence (two occurrences, same signature, same day)

Rank logs (docker logs --timestamps) around each exit:

rank 1: error: rank 1: a call passed its deadline (TF_TP_TIMEOUT_S): the other rank stalled; exiting
rank 0: error: rank 0: release not sent to rank 1 (FollowerGone)          (a handful, all on long prompts)
rank 0: error: rank 0: the other rank closed the control link (it exited, or the link is down); exiting

Both times the deadline fired ~300 s after a ~308k-token prompt arrived, and the failing prompt sizes differ
by single digits between runs (the same long agent sessions were replayed after each crash), i.e. this is
reproducible, not a flake. ExitCode=3, OOMKilled=false (the exit is abortAndExit's std.c._exit(3)).

Ruled out on the hardware side: clocks 2197 MHz at P0 with nvidia-smi -lgc effective (no low-power lock);
rank 0's log has no ≥5 s gap and keeps answering /health, /metrics, /v1/models with 200 until the last
second (the HTTP plane is fine — only the TP control plane goes silent); no OOM, no NVRM/Xid in either window.

Suggested fix

Make the follower's wait scale the same way the leader's allowance does — the follower can only know the size of
the call it is waiting on if the leader tells it (the prefill / prefill_many record already carries
prompt_len per stream), so either

• arm the follower's wait from the same sum(prompt_len)/400 value the leader uses, or
• have the leader emit a periodic progress/heartbeat record during a long call (the follower's deadline resets on
  every record, so a heartbeat alone fixes it without changing any timeout semantics).

The current coupling means the leader's allowed work and the follower's patience must be derived from the same
quantity; today they are not.

Workaround

TF_TP_TIMEOUT_S is read from the environment and reaches both ranks, so raising it (e.g. 900) lifts the
follower's wait and the leader's floor together. This is what we run now — it removes the failure but also widens
the window in which a genuinely dead peer is noticed.

Also noticed (minor, separate)

• /health live.decode_tokens_per_second / live.prefill_tokens_per_second read 0.0 for minutes at a time while
  the GPU is busy (measured: 6 requests running, generation_tokens_running=2733, 25/25 one-second samples read
  0.0), while live.connections / live.waiting update normally. Emitting the field only when it is known
  (or a short smoothing window) would let monitoring distinguish "idle" from "no reading".
• README.md "API notes" lists a /stats endpoint; on this build GET /stats returns
  404 {"error":{"message":"unknown path /stats"}}.
• (Might be worth documenting) start.sh's memory probe parses free, whose header is localized — under
  LANG=zh_CN.UTF-8 it reads an empty value and reports ERROR: only GiB memory available (note the missing
  number) with exit 1. Reading /proc/meminfo (or LC_ALL=C free) would make it locale-independent; we worked
  around it with an LC_ALL=C wrapper.