Skip to content

experiments_per_job: 1 launches duplicate processes on whole-node/exclusive allocations #177

Description

@Grutschus

srun fans out multiple processes if ntasks is not set explicitly

Issue

With experiments_per_job: 1, the sbatch template runs each experiment via a bare srun (no --ntasks). If cpus-per-task is set but a task count isn't, srun's default becomes floor(alloc_cpus / cpus_per_task). On a partition that grants more CPUs than requested (e.g. OverSubscribe=EXCLUSIVE / whole-node), one experiment is launched as N concurrent processes.

Cause

seml/commands/start.py (v0.5.6, ~L203):

srun_str = '' if experiments_per_job > 1 else 'srun '

This bare 'srun ' is substituted as {maybe_srun} into seml/templates/slurm/slurm_template.sh with no -n. Per SLURM manpage on srun:

If -c is specified without -n, as many tasks will be allocated per node as possible while satisfying the -c restriction.

Suggested fix

Pin one task in the single-experiment path, while still honoring an explicit user task count:

srun_str = ""
if experiments_per_job <= 1:
    _task_keys = ('ntasks', 'n', 'ntasks-per-node', 'ntasks-per-gpu', 'ntasks-per-socket')
    srun_str = 'srun ' if any(k in sbatch_options for k in _task_keys) else 'srun --ntasks=1 '

Workaround

(or why I am not submitting a PR rn)
Add ntasks: 1 to sbatch_options. Explicitly setting it will populate the corresponding env var, and srun will inherit the setting.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions