diff --git a/CHANGELOG.md b/CHANGELOG.md index 331d173..3434ec3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,51 @@ and this project adheres to ### Added +- Three more skills covering the things an agent hits in the first ten minutes + of real work on the cluster: + + `bodhi-storage` — `/beevol` is one shared BeeGFS mount and the compute + node's `/tmp` is a 423G local disk, so inputs are read from the former and + scratch written to the latter and cleaned up with a `trap`. Slurm hands out + no private temp directory here (`TMPDIR` is plain `/tmp`, `SLURM_TMPDIR` is + unset), which is why uncleaned job directories accumulate. It also records + that `du` on a home directory can run for minutes on BeeGFS, and that at 84% + full a large write is somebody else's problem too. + + `bodhi-software` — the order is module, then container, then `pixi`/`uv`. + The tree at `/cluster/software/modules-sw` carries around 137 packages, so + most of a genomics pipeline is a `module load` away and building it from + source is wasted time. Pin the version rather than taking `(default)`, load + inside the job script because `sbatch` starts from a clean login shell, and + note that `module avail` writes to stderr so grepping it needs `2>&1`. + + `slurm-batch` — for work that is per-sample rather than one command: + `sbatch` scripts, arrays throttled with `%N`, `--parsable` dependency + chains, and sizing the next run from `sacct`. Records the local numbers that + bite: `DefMemPerCPU` is 4000 MB so omitting `--mem` is not "unlimited", + `MaxArraySize` is 1001 so longer lists need chunking, `kill_invalid_depend` + is set so a dependent job vanishes rather than hangs when its upstream + fails, and `MaxRSS` is reported on the step rows where `sacct -X` will not + show it. + +- A `slurm-discovery` skill, for finding out what the cluster actually offers + instead of assuming it: what the partitions are and how big, which accounts + and QOS the user holds, and the rule that decides whether a combination is + submittable — your account in the partition's `AllowAccounts`, and the QOS + you ask for in both its `AllowQos` and your own association. That + intersection is the part nobody guesses right: on Bodhi the `gpu` partition + takes `gpu_rbi`/`gpu_devbio`/`gpu_scb` and not the default `rbi` account, so + the request is refused however many GPUs are idle, and the error names + neither half. It also covers reading the QOS limit columns, and `squeue`'s + reason column when a job is rejected or sits `PENDING`. + + The survey's answers are cached to + `~/.cache/sinteractive/slurm-map-.md` and re-read rather than + re-run. Keyed by `ClusterName` because one `$HOME` is often mounted on + several clusters, and a map from the wrong one is worse than none. Only the + structure is cached — node states and queue depth are re-read live every + time, so the cached `sinfo` deliberately drops the state column. + - A second Claude Code skill, `git-workflow`, installed alongside `bodhi-compute` by `--install-claude`. Where `bodhi-compute` is about the cluster, this one is about the repository open in the session: semantic diff --git a/Makefile b/Makefile index 35f9ef9..27a8aee 100644 --- a/Makefile +++ b/Makefile @@ -54,12 +54,18 @@ install-system: # --------------------------------------------------------------------------- # Claude Code integration. Two parts: # -# - skills, which teach an agent how work is done here. bodhi-compute -# covers cluster etiquette: the login node and an sinteractive session are -# both orchestration shells, and real work goes into its own allocation. -# git-workflow covers the git conventions — semantic versioning, -# Conventional Commits, a worktree per branch, landing through a pull -# request — and is about the repository in the session, not the cluster; +# - skills, which teach an agent how work is done here, loaded on demand +# from their descriptions rather than all at once: +# +# bodhi-compute cluster etiquette; a session is not a compute target +# slurm-discovery partitions, accounts, QOS, and what you may submit +# bodhi-storage /beevol vs node-local /tmp, and where output belongs +# bodhi-software modules first, then containers, then pixi/uv +# slurm-batch sbatch, arrays, dependencies, sizing from sacct +# git-workflow semver, Conventional Commits, worktrees, PRs +# +# The first five are about the cluster; git-workflow is about the +# repository open in the session; # - two hooks for an agent running INSIDE a session, which tell it at # startup where it is and how big the allocation is, and warn it when the # session is running out of walltime. diff --git a/README.md b/README.md index f2c08f7..5349a1e 100644 --- a/README.md +++ b/README.md @@ -279,12 +279,20 @@ and new panes, but shells already running keep their original `SINTERACTIVE_NAME`. > [!TIP] -> This repo ships two [Claude Code skills](https://code.claude.com/docs/en/skills) -> plus two hooks, for agents that run **inside** a session. `bodhi-compute` -> teaches cluster etiquette; `git-workflow` teaches the git conventions — -> semantic versioning, Conventional Commits, a worktree per branch, landing -> through a pull request. The hooks brief the agent on which session it is in -> at startup, and warn it when the session is running out of wall time. +> This repo ships six [Claude Code skills](https://code.claude.com/docs/en/skills) +> plus two hooks, for agents that run **inside** a session: +> +> | Skill | Teaches | +> |---|---| +> | `bodhi-compute` | Cluster etiquette — a session is not a compute target | +> | `slurm-discovery` | What the cluster offers: partitions, accounts, QOS, limits | +> | `bodhi-storage` | `/beevol` vs node-local `/tmp`, and where output belongs | +> | `bodhi-software` | Modules first, then containers, then pixi/uv | +> | `slurm-batch` | `sbatch`, arrays, dependencies, right-sizing from `sacct` | +> | `git-workflow` | Semver, Conventional Commits, worktrees, pull requests | +> +> The hooks brief the agent on which session it is in at startup, and warn it +> when the session is running out of wall time. > > ```bash > sinteractive --install-claude # from any installed copy diff --git a/docs/scripting.md b/docs/scripting.md index 3e49c8c..97f44c3 100644 --- a/docs/scripting.md +++ b/docs/scripting.md @@ -148,14 +148,41 @@ need them too, since running `--install-claude` from inside a session runs the node's copy of the script. Point `SINTERACTIVE_SHARE` at a checkout to override, and `make nodes-check` to see which nodes actually have them. -**Two [skills](https://code.claude.com/docs/en/skills)** teach agents how work -is done here. +**Six [skills](https://code.claude.com/docs/en/skills)** teach agents how work +is done here. Skills load on demand from their descriptions, so an agent picks +up the one the task calls for rather than carrying all six. `bodhi-compute` covers cluster etiquette: neither the login node nor an sinteractive session is a compute target, real work goes into an allocation sized for it, reuse sessions rather than piling them up, check the time budget before long jobs, and clean up. +`slurm-discovery` covers finding out what the cluster offers rather than +assuming it: what the partitions are and how big, which accounts and QOS you +hold, and the rule that decides whether a given combination is submittable — +your account in the partition's `AllowAccounts`, your QOS in both its +`AllowQos` and your own association. It also covers reading `squeue`'s reason +column when a job is refused or sits `PENDING`, and caches the answers per +cluster so the survey is run once rather than every session. + +`bodhi-storage` covers where data goes: `/beevol` is one shared BeeGFS mount +and the compute node's `/tmp` is a local disk, so inputs are read from the +former and scratch is written to the latter and cleaned up on exit. It also +warns that `du` on a home directory can run for minutes, and that the shared +filesystem is full enough for a large write to be somebody else's problem too. + +`bodhi-software` covers how to get a tool: the module tree first — around 137 +preinstalled packages, so most of a genomics pipeline is a `module load` away +— then a container, then `pixi`/`uv` for the remainder. Pin the version, +load inside the job script rather than the login shell, and never `pip +install` into the system Python. + +`slurm-batch` covers work that is per-sample rather than a single command: +`sbatch` scripts, arrays and why to throttle them with `%N`, dependency +chains, and using `sacct` to size the next run from what the last one actually +used — noting that `MaxRSS` lives on the step rows, where `sacct -X` will not +show it. + `git-workflow` covers the git conventions, and is about the repository open in the session rather than the cluster: semantic versioning with annotated `vX.Y.Z` tags, Conventional Commit messages, one worktree per branch under diff --git a/man/sinteractive.1 b/man/sinteractive.1 index 52dc870..9e36571 100644 --- a/man/sinteractive.1 +++ b/man/sinteractive.1 @@ -486,10 +486,18 @@ run from inside one create their own allocations rather than steps of the session's job. .SS Claude Code hooks .B \-\-agent\-context -prints this briefing for the current session. Two skills and two hooks wire +prints this briefing for the current session. Six skills and two hooks wire it up \(em .I bodhi\-compute -for cluster etiquette and +for cluster etiquette, +.I slurm\-discovery +for partitions, accounts and QOS, +.I bodhi\-storage +for where data belongs, +.I bodhi\-software +for modules and environments, +.I slurm\-batch +for array jobs and dependencies, and .I git\-workflow for the git conventions, plus one hook briefing the agent at session start and one warning it when the session is running out of wall time \(em diff --git a/skills/bodhi-compute/SKILL.md b/skills/bodhi-compute/SKILL.md index 812d4b5..fe13283 100644 --- a/skills/bodhi-compute/SKILL.md +++ b/skills/bodhi-compute/SKILL.md @@ -101,6 +101,11 @@ for sinteractive sessions and is the smallest partition on the cluster Request only what the task needs, and ask the user before requesting more than a day of walltime or a whole node's worth of resources. +`gpu` and some other partitions restrict which accounts and QOS may submit to +them, so the right `-p` can still be rejected under the wrong `-A`. The +`slurm-discovery` skill covers mapping that out, and reading the reason when a +job is refused or stuck `PENDING`. + ## sinteractive sessions These are the user's persistent interactive shells. You mostly *observe* them; @@ -210,3 +215,10 @@ ssh NODE "${SINTERACTIVE_TMUX:-/usr/local/bin/tmux}" -L sinteractive-JOBID \ Cancel allocations you created as soon as the work is done — `scancel ID` for an `salloc`, `sinteractive --cancel JOBID|NAME` for a session. Never cancel a session you did not create without asking the user. + +## Related skills + +- `slurm-discovery` — which partitions, accounts and QOS you may actually use. +- `bodhi-storage` — read from `/beevol`, scratch on node-local `/tmp`. +- `bodhi-software` — check `module avail` before building or installing. +- `slurm-batch` — `sbatch`, arrays and dependencies, when it is not one job. diff --git a/skills/bodhi-software/SKILL.md b/skills/bodhi-software/SKILL.md new file mode 100644 index 0000000..d334711 --- /dev/null +++ b/skills/bodhi-software/SKILL.md @@ -0,0 +1,79 @@ +--- +name: bodhi-software +description: How software is provided on the Bodhi cluster — the module tree of ~137 preinstalled bioinformatics tools, Singularity containers, and user-level pixi/uv environments — and the order to try them in. Use before installing, building, or compiling anything, when a command is not found, or when choosing how to pin a tool version for a pipeline. +--- + +# Getting a tool + +**Look before you build.** The cluster already ships around 137 packages, and +building `bcftools` from source or pip-installing a bioinformatics tool that +is one `module load` away wastes an hour and produces a less reproducible +result. + +The order is: **module → container → pixi/uv**. + +## 1. Modules + +Environment Modules 5.3.0 (Tcl, not Lmod — there is no `module spider`): + +```bash +module avail # the whole catalogue +module avail 2>&1 | grep -i star # avail writes to STDERR; grep needs 2>&1 +module whatis samtools # what a name resolves to +module show samtools # what it puts on PATH, and its own deps +module load samtools/1.22.1 +module list +module purge # start clean +``` + +The tree at `/cluster/software/modules-sw` covers most of what a genomics +pipeline needs — aligners (`bwa`, `bowtie2`, `STAR`, `minimap2`, `hisat2`), +`samtools`/`bcftools`/`htslib`/`bedtools`/`bedops`, `cellranger` and friends, +`picard`, `ncbi-blast`, `salmon`, `kallisto`, `R` (4.3.3, 4.5.1, 4.5.2), +`java` (8 through 25), `plink`, `sratoolkit`. + +**Pin the version.** `module load STAR` takes whatever carries `(default)` +today, and defaults move — `module load STAR/2.7.11b` is what makes a run +reproducible next year. Modules also load their own dependencies (`samtools` +pulls in `htslib`), so do not hand-assemble a stack that `module show` already +describes. + +**Load inside the job, not just the login shell.** A module loaded in your +session is an environment change; `sbatch` starts from a fresh login shell and +will not have it. Put the `module load` lines in the job script. + +## 2. Containers + +```bash +singularity exec /path/to/image.sif command ... +singularity exec --bind /beevol:/beevol image.sif command ... +``` + +Reach for this when a tool is not in the module tree and comes with an +official image, or when a pipeline pins one. Bind `/beevol` explicitly if the +tool needs to see cluster paths. + +## 3. pixi and uv + +For everything left over, and for project-local environments that belong to a +repository rather than to the cluster: + +```bash +pixi add samtools # project env, recorded in pixi.toml +pixi global install jq # a small tool you want on PATH everywhere +uv venv && uv pip install ... # Python projects +``` + +Both live under `$HOME`, which is shared across every node, so an environment +built once in a session works in every allocation without reinstalling. + +**Never `pip install` into the system Python.** It is not writable, and +`--user` puts packages on a path every job inherits, which turns one project's +pin into every project's problem. Use a project environment. + +## Building from source + +Only after checking `module avail`. If you do build, it is real compute — +give it its own allocation rather than running it in the session shell (see +`bodhi-compute`), install into `$HOME` or a project prefix, and write down in +the project what was built and why the module tree was not enough. diff --git a/skills/bodhi-storage/SKILL.md b/skills/bodhi-storage/SKILL.md new file mode 100644 index 0000000..fc9e7b8 --- /dev/null +++ b/skills/bodhi-storage/SKILL.md @@ -0,0 +1,86 @@ +--- +name: bodhi-storage +description: Where data lives on the Bodhi cluster and where to write it — the shared /beevol BeeGFS filesystem versus node-local /tmp, which one intermediates belong on, and how to check space without hanging the session. Use before writing large output, staging inputs, choosing a working directory for a job, or when a job is slow at I/O or the filesystem is full. +--- + +# Where does it go? + +## One shared filesystem, and one local disk + +```bash +df -h /beevol /tmp +# beegfs_nodev 839T 700T 139T 84% /beevol <- shared, everyone +# /dev/mapper/system-root 423G 22G 401G 6% / <- this node only +``` + +`/beevol` is a single BeeGFS mount and the only thing shared between nodes: + +| Path | What it is | +|---|---| +| `/beevol/home/$USER` | Your home. Code, environments, results worth keeping. | +| `/beevol/data` | Shared reference and project data. | +| `/beevol/illumina` | Sequencer output — `runs/`, `data/`. Read from it, don't write to it. | + +`/tmp` is the compute node's own 423G disk (Slurm's `TmpFS`), and `/dev/shm` +is a 377G tmpfs — RAM, so anything you put there counts against your job's +`--mem` and disappears with the allocation. + +**The filesystem is 84% full and it is shared with everyone.** Space you free +is space someone else's run does not fail for. + +## The rule + +**Read inputs from `/beevol`, write scratch to node-local `/tmp`, copy the +results back.** A pipeline that streams thousands of small writes to BeeGFS +is slow for you and slow for everyone else on the cluster; the same work +against local disk is not. + +Slurm does not hand out a private temp directory here — `TMPDIR` is plain +`/tmp` and `SLURM_TMPDIR` is unset — so make your own and clean it up, +because nothing else will: + +```bash +#!/usr/bin/env bash +set -euo pipefail +work=/tmp/$USER-$SLURM_JOB_ID +mkdir -p "$work" +trap 'rm -rf "$work"' EXIT # runs on success, failure, and scancel + +samtools sort -@ 8 -T "$work"/sort -o "$work"/out.bam /beevol/data/in.bam +cp "$work"/out.bam /beevol/home/$USER/results/ +``` + +The `trap` matters: `/tmp` on a shared node already has a couple of thousand +entries, and an uncleaned job directory sits there until someone notices. + +Keep the *final* artifacts on `/beevol` — `/tmp` is node-local, so the next +job in the pipeline probably lands somewhere else and cannot see it. + +## Checking space + +`df` is instant. **`du` on a home directory is not** — BeeGFS has to walk +every file, and `du -sh /beevol/home/$USER` can run for many minutes and give +you nothing to show for it. Point it at a subdirectory you actually suspect, +give it a timeout, and run it in an allocation rather than in the session: + +```bash +df -h /beevol # always safe +du -sh --max-depth=1 ~/devel 2>/dev/null # one level, one subtree +``` + +For finding what to delete, target the big and the old rather than +summarising everything: + +```bash +find ~/ -xdev -type f -size +5G -printf '%s\t%p\n' 2>/dev/null | sort -rn | head +``` + +## Before writing something large + +Estimate the output, check `df -h /beevol`, and say so if the run would take a +visible bite out of the remaining 139T. Ask the user before writing hundreds +of gigabytes to shared storage — on a filesystem this full that is a decision +about other people's work, not just theirs. + +Sizing an allocation for the job that does the writing is the `bodhi-compute` +skill; finding out which partition you may use is `slurm-discovery`. diff --git a/skills/slurm-batch/SKILL.md b/skills/slurm-batch/SKILL.md new file mode 100644 index 0000000..d7a3d92 --- /dev/null +++ b/skills/slurm-batch/SKILL.md @@ -0,0 +1,116 @@ +--- +name: slurm-batch +description: Running many jobs with sbatch — job scripts, array jobs and their throttling, dependencies between stages, and using sacct to right-size the next run from what the last one actually used. Use when the work is per-sample or per-file rather than a single command, when building a multi-stage pipeline, or when sizing memory and walltime for a batch. +--- + +# Many jobs, not one + +`srun` and `salloc` (the `bodhi-compute` skill) are for one thing at a time, +attached. A batch of hundreds of samples is a different shape: it is submitted +and left, and its sizing is decided once and then applied hundreds of times — +which is what makes getting the sizing right worth a few minutes up front. + +## A job script + +```bash +#!/usr/bin/env bash +#SBATCH --job-name=star-align +#SBATCH --comment=star-align +#SBATCH --partition=rna +#SBATCH --account=rbi +#SBATCH --cpus-per-task=8 +#SBATCH --mem=32G +#SBATCH --time=4:00:00 +#SBATCH --output=logs/%x-%j.out +#SBATCH --error=logs/%x-%j.err +set -euo pipefail + +module load STAR/2.7.11b # sbatch starts from a clean login shell +... +``` + +Name it in **both** `--job-name` and `--comment`, the same short descriptive +value, for the same reason as `srun` — a shared queue should say what is +running and why. + +`--output` directories are not created for you: `mkdir -p logs` first, or the +job dies at startup with nowhere to write. + +**`DefMemPerCPU` is 4000 MB here.** Leaving `--mem` off does not mean +"unlimited", it means 4G per CPU — a frequent and confusing cause of a job +being killed for memory it never asked for. + +## Arrays + +One submission, one task per sample: + +```bash +#SBATCH --array=0-499%20 # 500 tasks, at most 20 running at once +sample=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" samples.txt) +``` + +- **`MaxArraySize` is 1001** on this cluster, so indices run `0-1000` and a + list longer than that has to be chunked into several submissions. +- **Throttle with `%N`.** Without it, 500 tasks all become eligible at once, + which fills the partition and pushes everyone else — including your own + later stages — behind them. `%20` is a courteous default; raise it when the + partition is idle. +- `%A` is the array job id and `%a` the task id, so + `--output=logs/%x-%A_%a.out` keeps per-task logs apart. `%j` alone collides. +- `MaxJobCount` is 10000 cluster-wide, and the QOS caps submissions per user + (`normal` allows 2000 submitted, 500 running). `slurm-discovery` covers + reading those. + +## Dependencies + +```bash +a=$(sbatch --parsable align.sh) +b=$(sbatch --parsable --dependency=afterok:$a merge.sh) +sbatch --dependency=afterok:$b report.sh +``` + +`--parsable` prints the bare job id, which is what makes this chainable. +`afterok` waits for success; `afterany` runs regardless; `singleton` serialises +jobs sharing a name. + +This cluster runs with `kill_invalid_depend`, so a dependency that can never +be satisfied — the job it waits on failed — is **killed rather than left +pending forever**. A vanished downstream job usually means an upstream +failure, so check that first with `sacct` rather than resubmitting. + +## Right-size from what actually happened + +Run one sample, then look at what it used before committing to hundreds: + +```bash +sacct -j JOBID --format=JobID,JobName%14,ReqMem,MaxRSS,AllocCPUS,Elapsed,State +``` + +**`MaxRSS` is reported on the step rows, not the allocation row.** The parent +line is blank and `sacct -X`, which shows allocations only, hides it entirely: + +``` +237176 bash 8G 4 00:00:36 COMPLETED +237176.0 bash 631852K 4 00:00:36 COMPLETED <- here +``` + +That job asked for 8G and touched 617M. Ask for what the measurement says plus +headroom, not a round number that felt safe — over-requesting memory and +walltime is what makes a queue slow for everybody, since the scheduler must +find a hole big enough for the request rather than the job. + +Walltime is the same trade in reverse: too short and the job is killed at the +limit with its output half-written, too long and backfill will not slot it in. +`Elapsed` from the trial run is the number to build on. + +## Watching and cleaning up + +```bash +squeue --me -o "%.10i %.20j %.10T %.10M %r" # %r is the reason if pending +scancel JOBID # whole array +scancel JOBID_7 # one task +scancel --me --name=star-align # by name +``` + +Cancel a batch you have abandoned rather than leaving it to drain the +partition. Never cancel jobs you did not submit. diff --git a/skills/slurm-discovery/SKILL.md b/skills/slurm-discovery/SKILL.md new file mode 100644 index 0000000..97a13e5 --- /dev/null +++ b/skills/slurm-discovery/SKILL.md @@ -0,0 +1,174 @@ +--- +name: slurm-discovery +description: Find out what a Slurm cluster actually offers you — which partitions exist and how big they are, which accounts and QOS you hold, and which combinations you are allowed to submit. Use when choosing a partition or QOS, when sizing a job against the limits, when a submission is rejected, or when a job sits PENDING and the reason is unclear. +--- + +# What can I actually run here? + +Ask the cluster rather than assuming. A handful of commands answer almost +everything, none costs more than a scheduler round-trip, and the answers that +matter are worth writing down once instead of rediscovering every session. + +## Check the cached map first + +Partitions, accounts and QOS change on the order of months. Once the survey +below has run, its answers live in a file — **read that before running +anything**. + +The file is keyed by cluster, because the same `$HOME` is often mounted on +more than one and a map from the wrong one is worse than none: + +```bash +cluster=$(scontrol show config | sed -n 's/^ClusterName *= *//p' | tr -d '[:space:]') +map=~/.cache/sinteractive/slurm-map-${cluster:-unknown}.md +cat "$map" # nothing? build it, below +``` + +Build or refresh it in one go. `sinteractive` only ever removes files named +after its own job ids, so this one is safe alongside them: + +```bash +mkdir -p ~/.cache/sinteractive +{ + echo "# Slurm map for $USER on ${cluster:-unknown}" + echo "# Generated $(date -Is). Rebuild when a partition or account changes." + echo; echo '## Partitions' + sinfo -o "%20P %5a %10l %10L %6D %8c %10m %12G" + echo; echo '## My associations (account|partition|QOS|...)' + sacctmgr -nP show assoc user="$USER" \ + format=Account,Partition,QOS,MaxJobs,MaxSubmit,GrpTRES,MaxTRES,MaxWall + echo; echo "## My default account" + sacctmgr -nP show user "$USER" format=User,DefaultAccount + echo; echo '## QOS limits' + sacctmgr -nP show qos \ + format=Name,Priority,MaxWall,MaxTRESPU,MaxJobsPU,MaxSubmitJobsPU,GrpTRES,Flags + echo; echo '## Partition access (AllowAccounts / AllowQos)' + scontrol show partition | grep -E '^PartitionName=|AllowAccounts=' +} > "$map" +``` + +Rebuild it when something stops matching — a partition you were told about is +missing, or an account is rejected that the map says you hold — and when the +`date -Is` in the header is more than a month or two old. + +**Cache the map, never the weather.** Node states, queue depth and who is +running what change by the minute, and a cached `idle` is a lie within +minutes. That is why the `sinfo` line above drops the state column the survey +below keeps: what belongs in the file is the structure — which partitions +exist, how big they are, what you may ask for, and the limits on it. Anything +about right now gets run live, every time. + +## The survey + +**What partitions exist, and how big are they?** + +```bash +sinfo -o "%20P %5a %10l %10L %6D %6t %8c %10m %12G %N" +# PARTITION AVAIL TIMELIMIT DEFAULTTIME NODES STATE CPUS MEMORY GRES NODELIST +``` + +`TIMELIMIT` is the ceiling, `DEFAULTTIME` is what you get by leaving `-t` +off — usually much shorter, and a common cause of a job dying early. `MEMORY` +is per node in MB, and a trailing `+` means the nodes in that row differ. + +**What do I hold?** + +```bash +sacctmgr -nP show assoc user=$USER \ + format=Account,Partition,QOS,MaxJobs,MaxSubmit,GrpTRES,MaxTRES,MaxWall +# rbi||high,long,normal,positron||||| +# gpu_rbi||high,long,normal||||| + +sacctmgr -nP show user $USER format=User,DefaultAccount +# jhessel|rbi +``` + +One row per account. The QOS column is what that account may request; empty +limit columns mean the limit comes from the QOS, not the association. + +**What does the partition allow?** + +```bash +scontrol show partition rna +# AllowGroups=ALL AllowAccounts=rbi AllowQos=long,normal +# DefaultTime=04:00:00 MaxTime=UNLIMITED DefMemPerNode=12000 +``` + +## The rule + +**You can submit to a partition when your account is in its `AllowAccounts` +and the QOS you ask for is in both its `AllowQos` and your association's QOS +list.** Both halves have to hold. Neither is implied by the other, and the +error you get for failing either is the same unhelpful "invalid account or +partition". + +The default account is the trap. On Bodhi, `gpu` has +`AllowAccounts=gpu_rbi,gpu_devbio,gpu_scb`, so a default account of `rbi` is +rejected there however many GPUs are idle — the fix is `-A gpu_rbi`, not a +smaller request: + +```bash +srun -p gpu -A gpu_rbi --gres=gpu:1 -c 8 --mem 32G -t 2:00:00 \ + -J probe --comment=probe -- nvidia-smi -L +``` + +## Reading the limits + +```bash +sacctmgr -nP show qos \ + format=Name,Priority,MaxWall,MaxTRESPU,MaxJobsPU,MaxSubmitJobsPU,GrpTRES,Flags +# normal|25|3-00:00:00||500|2000||DenyOnLimit +# long|50|7-00:00:00|cpu=128|12|50|cpu=156|OverPartQOS +# interactive|50|12:00:00|cpu=16,mem=8G|4|3||DenyOnLimit,OverPartQOS +``` + +- `MaxWall` caps a single job. Asking for more is rejected outright, not + trimmed — `long` is how you get past the `normal` QOS's ceiling. +- `MaxTRESPU` / `MaxJobsPU` are **per user**, and `GrpTRES` is across everyone + on that QOS. A job can be legal on its own and still queue because your + other jobs are already holding the budget. +- `OverPartQOS` means the QOS limit wins over the partition's; without it the + tighter of the two applies. +- `DenyOnLimit` rejects an over-limit job at submit time instead of queueing + it forever. Its absence is why some requests vanish into `PENDING`. + +## What is free right now + +```bash +sinfo -p rna -o "%6t %6D %8c %10m %N" # idle vs mix vs alloc, by node +squeue -p rna -o "%.10i %.10u %.10M %.6C %R" | head +``` + +`idle` nodes are whole and free; `mix` has room but is shared. Sizing a +request to what is actually idle is the difference between starting now and +starting tomorrow. + +## When a job will not run + +```bash +squeue --me -o "%.10i %.20j %.10T %r" # %r is the reason +``` + +The reason names the wall you hit: + +| Reason | Meaning | +|---|---| +| `Resources` | The request is legal; the nodes are busy. Wait, or shrink it. | +| `Priority` | Legal, but others are ahead. Check `sshare -U` for fairshare. | +| `QOSMaxWallDurationPerJobLimit` | `-t` exceeds the QOS `MaxWall`. Ask for a longer QOS. | +| `QOSMaxCpuPerUserLimit`, `AssocMaxJobsLimit` | Your own running jobs are holding the budget. | +| `PartitionTimeLimit`, `PartitionConfig` | The request cannot fit the partition at all. | +| `ReqNodeNotAvail` | Named nodes are down or drained — check `sinfo -R`. | + +For a job already rejected at submit, re-run with `--test-only` to get the +verdict without queueing anything: + +```bash +srun --test-only -p rna -A rbi -c 8 --mem 32G -t 1:00:00 -- true +``` + +## Then go run something + +This skill is about finding out what is available. Actually placing work — in +its own allocation, named in both `-J` and `--comment`, never in the shell you +are typing in — is the `bodhi-compute` skill.