Skip to content

worker: expose --heartbeat through hq alloc add#1117

Open
MauriceDHanisch wants to merge 1 commit into
It4innovations:mainfrom
MauriceDHanisch:fix/expose-heartbeat-passthrough
Open

worker: expose --heartbeat through hq alloc add#1117
MauriceDHanisch wants to merge 1 commit into
It4innovations:mainfrom
MauriceDHanisch:fix/expose-heartbeat-passthrough

Conversation

@MauriceDHanisch

Copy link
Copy Markdown

Summary

  • hq worker start already accepted --heartbeat, but hq alloc add slurm/hq alloc add pbs had no way to set it — the field lived on WorkerStartOpts, not the SharedWorkerStartOpts that autoalloc's generated worker command reuses.
  • Moves heartbeat onto SharedWorkerStartOpts (Option<Duration>, no default so we can detect "not set") and forwards it into the generated hq worker start ... invocation in autoalloc.rs only when explicitly provided, preserving the existing 8s default otherwise.

Motivation

On a busy server (e.g. mid MILP scheduler solve, or handling a burst of other workers registering), the single-threaded executor can go long enough without polling that an already-connected worker's heartbeat looks lost even though nothing is actually wrong. Autoalloc-managed workers had no way to raise their heartbeat interval to ride this out, since the flag was never wired through hq alloc add. Plain hq worker start was unaffected — this only closes the gap for the autoalloc path.

Test plan

  • cargo check -p hyperqueue -p tako
  • cargo test -p tako (no regressions)
  • Manually verified hq alloc add slurm --heartbeat 45s renders --heartbeat "45s" in the generated sbatch worker command

heartbeat lived only on WorkerStartOpts (plain 'hq worker start'), not on
SharedWorkerStartOpts, which alloc add slurm/pbs reuses for pilot worker
args. Autoalloc-managed campaigns (100% of production usage) had no way to
raise the default 8s heartbeat interval, which is too tight under the
single-threaded executor's scheduling/registration load (see It4innovations#7).

Move heartbeat onto SharedWorkerStartOpts as Option<Duration> with the 8s
default applied at WorkerConfiguration construction instead of via clap's
default_value, so plain worker start behavior is unchanged and alloc add
can now pass --heartbeat through to the generated worker start command.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant