From 6706bd0d7c475695452344f3a07fb3e8cb416ea3 Mon Sep 17 00:00:00 2001 From: Jay Hesselberth Date: Wed, 26 Aug 2026 06:21:17 -0600 Subject: [PATCH 1/2] feat(claude): add a slurm-discovery skill with a per-cluster cached map MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bodhi-compute tells an agent to pick a partition with sinfo, which is only half the story: partitions restrict which accounts and QOS may submit to them, and the request has to satisfy both the partition's AllowAccounts/AllowQos and the user's own association. Neither half implies the other, and failing either produces the same unhelpful "invalid account or partition". That intersection is the part nobody guesses right. On Bodhi the gpu partition takes gpu_rbi/gpu_devbio/gpu_scb and not the default rbi account, so the job is refused however many GPUs are idle, and the error names neither the account nor the QOS. An agent that does not know to look ends up shrinking a request that was never too big. The skill covers the survey, the rule, reading the QOS limit columns, and squeue's reason column when a job is rejected or stuck PENDING. Every command in it was run against Bodhi rather than recalled. Its answers are cached to ~/.cache/sinteractive/slurm-map-.md and re-read rather than re-run, since partitions and accounts change on the order of months. The file is keyed by ClusterName because one $HOME is frequently mounted on several clusters and a map from the wrong one is worse than none, and it lives beside the session state files, which sinteractive only ever removes by job id. Only the structure is cached: node state and queue depth are re-read live every time, which is why the cached sinfo deliberately drops the state column a cached "idle" would misreport within minutes. No installer change was needed — skills are discovered from what ships beside the script. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 18 ++++ Makefile | 9 +- README.md | 6 +- docs/scripting.md | 11 +- man/sinteractive.1 | 6 +- skills/bodhi-compute/SKILL.md | 5 + skills/slurm-discovery/SKILL.md | 174 ++++++++++++++++++++++++++++++++ 7 files changed, 220 insertions(+), 9 deletions(-) create mode 100644 skills/slurm-discovery/SKILL.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 331d173..3796ea7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,24 @@ and this project adheres to ### Added +- A `slurm-discovery` skill, for finding out what the cluster actually offers + instead of assuming it: what the partitions are and how big, which accounts + and QOS the user holds, and the rule that decides whether a combination is + submittable — your account in the partition's `AllowAccounts`, and the QOS + you ask for in both its `AllowQos` and your own association. That + intersection is the part nobody guesses right: on Bodhi the `gpu` partition + takes `gpu_rbi`/`gpu_devbio`/`gpu_scb` and not the default `rbi` account, so + the request is refused however many GPUs are idle, and the error names + neither half. It also covers reading the QOS limit columns, and `squeue`'s + reason column when a job is rejected or sits `PENDING`. + + The survey's answers are cached to + `~/.cache/sinteractive/slurm-map-.md` and re-read rather than + re-run. Keyed by `ClusterName` because one `$HOME` is often mounted on + several clusters, and a map from the wrong one is worse than none. Only the + structure is cached — node states and queue depth are re-read live every + time, so the cached `sinfo` deliberately drops the state column. + - A second Claude Code skill, `git-workflow`, installed alongside `bodhi-compute` by `--install-claude`. Where `bodhi-compute` is about the cluster, this one is about the repository open in the session: semantic diff --git a/Makefile b/Makefile index 35f9ef9..4d49af6 100644 --- a/Makefile +++ b/Makefile @@ -57,9 +57,12 @@ install-system: # - skills, which teach an agent how work is done here. bodhi-compute # covers cluster etiquette: the login node and an sinteractive session are # both orchestration shells, and real work goes into its own allocation. -# git-workflow covers the git conventions — semantic versioning, -# Conventional Commits, a worktree per branch, landing through a pull -# request — and is about the repository in the session, not the cluster; +# slurm-discovery covers finding out what the cluster offers — partitions, +# accounts, QOS and which combinations submit — and caching that map per +# cluster rather than rediscovering it. git-workflow covers the git +# conventions — semantic versioning, Conventional Commits, a worktree per +# branch, landing through a pull request — and is about the repository in +# the session, not the cluster; # - two hooks for an agent running INSIDE a session, which tell it at # startup where it is and how big the allocation is, and warn it when the # session is running out of walltime. diff --git a/README.md b/README.md index f2c08f7..f32325e 100644 --- a/README.md +++ b/README.md @@ -279,9 +279,11 @@ and new panes, but shells already running keep their original `SINTERACTIVE_NAME`. > [!TIP] -> This repo ships two [Claude Code skills](https://code.claude.com/docs/en/skills) +> This repo ships three [Claude Code skills](https://code.claude.com/docs/en/skills) > plus two hooks, for agents that run **inside** a session. `bodhi-compute` -> teaches cluster etiquette; `git-workflow` teaches the git conventions — +> teaches cluster etiquette; `slurm-discovery` teaches how to find out what +> the cluster actually offers you — partitions, accounts, QOS, and which +> combinations you may submit; `git-workflow` teaches the git conventions — > semantic versioning, Conventional Commits, a worktree per branch, landing > through a pull request. The hooks brief the agent on which session it is in > at startup, and warn it when the session is running out of wall time. diff --git a/docs/scripting.md b/docs/scripting.md index 3e49c8c..50b1df5 100644 --- a/docs/scripting.md +++ b/docs/scripting.md @@ -148,14 +148,21 @@ need them too, since running `--install-claude` from inside a session runs the node's copy of the script. Point `SINTERACTIVE_SHARE` at a checkout to override, and `make nodes-check` to see which nodes actually have them. -**Two [skills](https://code.claude.com/docs/en/skills)** teach agents how work -is done here. +**Three [skills](https://code.claude.com/docs/en/skills)** teach agents how +work is done here. `bodhi-compute` covers cluster etiquette: neither the login node nor an sinteractive session is a compute target, real work goes into an allocation sized for it, reuse sessions rather than piling them up, check the time budget before long jobs, and clean up. +`slurm-discovery` covers finding out what the cluster offers rather than +assuming it: what the partitions are and how big, which accounts and QOS you +hold, and the rule that decides whether a given combination is submittable — +your account in the partition's `AllowAccounts`, your QOS in both its +`AllowQos` and your own association. It also covers reading `squeue`'s reason +column when a job is refused or sits `PENDING`. + `git-workflow` covers the git conventions, and is about the repository open in the session rather than the cluster: semantic versioning with annotated `vX.Y.Z` tags, Conventional Commit messages, one worktree per branch under diff --git a/man/sinteractive.1 b/man/sinteractive.1 index 52dc870..5fa9efe 100644 --- a/man/sinteractive.1 +++ b/man/sinteractive.1 @@ -486,10 +486,12 @@ run from inside one create their own allocations rather than steps of the session's job. .SS Claude Code hooks .B \-\-agent\-context -prints this briefing for the current session. Two skills and two hooks wire +prints this briefing for the current session. Three skills and two hooks wire it up \(em .I bodhi\-compute -for cluster etiquette and +for cluster etiquette, +.I slurm\-discovery +for mapping partitions, accounts and QOS, and .I git\-workflow for the git conventions, plus one hook briefing the agent at session start and one warning it when the session is running out of wall time \(em diff --git a/skills/bodhi-compute/SKILL.md b/skills/bodhi-compute/SKILL.md index 812d4b5..a6fb9c9 100644 --- a/skills/bodhi-compute/SKILL.md +++ b/skills/bodhi-compute/SKILL.md @@ -101,6 +101,11 @@ for sinteractive sessions and is the smallest partition on the cluster Request only what the task needs, and ask the user before requesting more than a day of walltime or a whole node's worth of resources. +`gpu` and some other partitions restrict which accounts and QOS may submit to +them, so the right `-p` can still be rejected under the wrong `-A`. The +`slurm-discovery` skill covers mapping that out, and reading the reason when a +job is refused or stuck `PENDING`. + ## sinteractive sessions These are the user's persistent interactive shells. You mostly *observe* them; diff --git a/skills/slurm-discovery/SKILL.md b/skills/slurm-discovery/SKILL.md new file mode 100644 index 0000000..97a13e5 --- /dev/null +++ b/skills/slurm-discovery/SKILL.md @@ -0,0 +1,174 @@ +--- +name: slurm-discovery +description: Find out what a Slurm cluster actually offers you — which partitions exist and how big they are, which accounts and QOS you hold, and which combinations you are allowed to submit. Use when choosing a partition or QOS, when sizing a job against the limits, when a submission is rejected, or when a job sits PENDING and the reason is unclear. +--- + +# What can I actually run here? + +Ask the cluster rather than assuming. A handful of commands answer almost +everything, none costs more than a scheduler round-trip, and the answers that +matter are worth writing down once instead of rediscovering every session. + +## Check the cached map first + +Partitions, accounts and QOS change on the order of months. Once the survey +below has run, its answers live in a file — **read that before running +anything**. + +The file is keyed by cluster, because the same `$HOME` is often mounted on +more than one and a map from the wrong one is worse than none: + +```bash +cluster=$(scontrol show config | sed -n 's/^ClusterName *= *//p' | tr -d '[:space:]') +map=~/.cache/sinteractive/slurm-map-${cluster:-unknown}.md +cat "$map" # nothing? build it, below +``` + +Build or refresh it in one go. `sinteractive` only ever removes files named +after its own job ids, so this one is safe alongside them: + +```bash +mkdir -p ~/.cache/sinteractive +{ + echo "# Slurm map for $USER on ${cluster:-unknown}" + echo "# Generated $(date -Is). Rebuild when a partition or account changes." + echo; echo '## Partitions' + sinfo -o "%20P %5a %10l %10L %6D %8c %10m %12G" + echo; echo '## My associations (account|partition|QOS|...)' + sacctmgr -nP show assoc user="$USER" \ + format=Account,Partition,QOS,MaxJobs,MaxSubmit,GrpTRES,MaxTRES,MaxWall + echo; echo "## My default account" + sacctmgr -nP show user "$USER" format=User,DefaultAccount + echo; echo '## QOS limits' + sacctmgr -nP show qos \ + format=Name,Priority,MaxWall,MaxTRESPU,MaxJobsPU,MaxSubmitJobsPU,GrpTRES,Flags + echo; echo '## Partition access (AllowAccounts / AllowQos)' + scontrol show partition | grep -E '^PartitionName=|AllowAccounts=' +} > "$map" +``` + +Rebuild it when something stops matching — a partition you were told about is +missing, or an account is rejected that the map says you hold — and when the +`date -Is` in the header is more than a month or two old. + +**Cache the map, never the weather.** Node states, queue depth and who is +running what change by the minute, and a cached `idle` is a lie within +minutes. That is why the `sinfo` line above drops the state column the survey +below keeps: what belongs in the file is the structure — which partitions +exist, how big they are, what you may ask for, and the limits on it. Anything +about right now gets run live, every time. + +## The survey + +**What partitions exist, and how big are they?** + +```bash +sinfo -o "%20P %5a %10l %10L %6D %6t %8c %10m %12G %N" +# PARTITION AVAIL TIMELIMIT DEFAULTTIME NODES STATE CPUS MEMORY GRES NODELIST +``` + +`TIMELIMIT` is the ceiling, `DEFAULTTIME` is what you get by leaving `-t` +off — usually much shorter, and a common cause of a job dying early. `MEMORY` +is per node in MB, and a trailing `+` means the nodes in that row differ. + +**What do I hold?** + +```bash +sacctmgr -nP show assoc user=$USER \ + format=Account,Partition,QOS,MaxJobs,MaxSubmit,GrpTRES,MaxTRES,MaxWall +# rbi||high,long,normal,positron||||| +# gpu_rbi||high,long,normal||||| + +sacctmgr -nP show user $USER format=User,DefaultAccount +# jhessel|rbi +``` + +One row per account. The QOS column is what that account may request; empty +limit columns mean the limit comes from the QOS, not the association. + +**What does the partition allow?** + +```bash +scontrol show partition rna +# AllowGroups=ALL AllowAccounts=rbi AllowQos=long,normal +# DefaultTime=04:00:00 MaxTime=UNLIMITED DefMemPerNode=12000 +``` + +## The rule + +**You can submit to a partition when your account is in its `AllowAccounts` +and the QOS you ask for is in both its `AllowQos` and your association's QOS +list.** Both halves have to hold. Neither is implied by the other, and the +error you get for failing either is the same unhelpful "invalid account or +partition". + +The default account is the trap. On Bodhi, `gpu` has +`AllowAccounts=gpu_rbi,gpu_devbio,gpu_scb`, so a default account of `rbi` is +rejected there however many GPUs are idle — the fix is `-A gpu_rbi`, not a +smaller request: + +```bash +srun -p gpu -A gpu_rbi --gres=gpu:1 -c 8 --mem 32G -t 2:00:00 \ + -J probe --comment=probe -- nvidia-smi -L +``` + +## Reading the limits + +```bash +sacctmgr -nP show qos \ + format=Name,Priority,MaxWall,MaxTRESPU,MaxJobsPU,MaxSubmitJobsPU,GrpTRES,Flags +# normal|25|3-00:00:00||500|2000||DenyOnLimit +# long|50|7-00:00:00|cpu=128|12|50|cpu=156|OverPartQOS +# interactive|50|12:00:00|cpu=16,mem=8G|4|3||DenyOnLimit,OverPartQOS +``` + +- `MaxWall` caps a single job. Asking for more is rejected outright, not + trimmed — `long` is how you get past the `normal` QOS's ceiling. +- `MaxTRESPU` / `MaxJobsPU` are **per user**, and `GrpTRES` is across everyone + on that QOS. A job can be legal on its own and still queue because your + other jobs are already holding the budget. +- `OverPartQOS` means the QOS limit wins over the partition's; without it the + tighter of the two applies. +- `DenyOnLimit` rejects an over-limit job at submit time instead of queueing + it forever. Its absence is why some requests vanish into `PENDING`. + +## What is free right now + +```bash +sinfo -p rna -o "%6t %6D %8c %10m %N" # idle vs mix vs alloc, by node +squeue -p rna -o "%.10i %.10u %.10M %.6C %R" | head +``` + +`idle` nodes are whole and free; `mix` has room but is shared. Sizing a +request to what is actually idle is the difference between starting now and +starting tomorrow. + +## When a job will not run + +```bash +squeue --me -o "%.10i %.20j %.10T %r" # %r is the reason +``` + +The reason names the wall you hit: + +| Reason | Meaning | +|---|---| +| `Resources` | The request is legal; the nodes are busy. Wait, or shrink it. | +| `Priority` | Legal, but others are ahead. Check `sshare -U` for fairshare. | +| `QOSMaxWallDurationPerJobLimit` | `-t` exceeds the QOS `MaxWall`. Ask for a longer QOS. | +| `QOSMaxCpuPerUserLimit`, `AssocMaxJobsLimit` | Your own running jobs are holding the budget. | +| `PartitionTimeLimit`, `PartitionConfig` | The request cannot fit the partition at all. | +| `ReqNodeNotAvail` | Named nodes are down or drained — check `sinfo -R`. | + +For a job already rejected at submit, re-run with `--test-only` to get the +verdict without queueing anything: + +```bash +srun --test-only -p rna -A rbi -c 8 --mem 32G -t 1:00:00 -- true +``` + +## Then go run something + +This skill is about finding out what is available. Actually placing work — in +its own allocation, named in both `-J` and `--comment`, never in the shell you +are typing in — is the `bodhi-compute` skill. From 2fd2e261657cf0c50a6ed77c32f8769a5a38e887 Mon Sep 17 00:00:00 2001 From: Jay Hesselberth Date: Wed, 26 Aug 2026 06:31:52 -0600 Subject: [PATCH 2/2] feat(claude): add bodhi-storage, bodhi-software and slurm-batch skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit slurm-discovery told an agent which partition it may use. These cover the next three things it hits, all of which are cheap to get wrong and invisible until they are: where the output goes, where the tool comes from, and how to run the same command four hundred times. bodhi-storage. /beevol is a single BeeGFS mount shared by everyone and 84% full; the compute node's /tmp is a 423G local disk. Inputs get read from the former and scratch written to the latter, then copied back — a pipeline streaming small writes to BeeGFS is slow for its owner and slow for everyone else. Slurm hands out no private temp directory here (TMPDIR is plain /tmp, SLURM_TMPDIR is unset), so the skill shows making one under the job id and clearing it with a trap, which is why /tmp on a shared node is already two thousand entries deep. It also records that du on a home directory can run for minutes on BeeGFS — long enough to look like a hang. bodhi-software. The order is module, then container, then pixi/uv. The tree at /cluster/software/modules-sw carries around 137 packages, so most of a genomics pipeline is one module load away and building it from source is an hour spent on a less reproducible result. Pin the version rather than taking whatever carries (default) today; load inside the job script, since sbatch starts from a clean login shell; and note that module avail writes to stderr, so grepping it without 2>&1 silently returns nothing. slurm-batch. For work that is per-sample rather than one command: job scripts, arrays throttled with %N so one submission does not fill the partition ahead of its own later stages, --parsable dependency chains, and sizing the next run from what the last one measured. The local numbers that bite are written down rather than left to be discovered: DefMemPerCPU is 4000 MB, so omitting --mem is not "unlimited"; MaxArraySize is 1001, so longer sample lists need chunking; kill_invalid_depend is set, so a dependent job vanishes rather than hangs when its upstream fails; and MaxRSS is reported on the step rows, where sacct -X will not show it at all. Every command and every number in the three was checked against Bodhi rather than recalled. bodhi-compute gains a related-skills list, and the docs, man page and Makefile comment now describe six skills. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 27 ++++++++ Makefile | 21 +++--- README.md | 22 ++++--- docs/scripting.md | 26 +++++++- man/sinteractive.1 | 10 ++- skills/bodhi-compute/SKILL.md | 7 ++ skills/bodhi-software/SKILL.md | 79 ++++++++++++++++++++++ skills/bodhi-storage/SKILL.md | 86 ++++++++++++++++++++++++ skills/slurm-batch/SKILL.md | 116 +++++++++++++++++++++++++++++++++ 9 files changed, 372 insertions(+), 22 deletions(-) create mode 100644 skills/bodhi-software/SKILL.md create mode 100644 skills/bodhi-storage/SKILL.md create mode 100644 skills/slurm-batch/SKILL.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 3796ea7..3434ec3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,33 @@ and this project adheres to ### Added +- Three more skills covering the things an agent hits in the first ten minutes + of real work on the cluster: + + `bodhi-storage` — `/beevol` is one shared BeeGFS mount and the compute + node's `/tmp` is a 423G local disk, so inputs are read from the former and + scratch written to the latter and cleaned up with a `trap`. Slurm hands out + no private temp directory here (`TMPDIR` is plain `/tmp`, `SLURM_TMPDIR` is + unset), which is why uncleaned job directories accumulate. It also records + that `du` on a home directory can run for minutes on BeeGFS, and that at 84% + full a large write is somebody else's problem too. + + `bodhi-software` — the order is module, then container, then `pixi`/`uv`. + The tree at `/cluster/software/modules-sw` carries around 137 packages, so + most of a genomics pipeline is a `module load` away and building it from + source is wasted time. Pin the version rather than taking `(default)`, load + inside the job script because `sbatch` starts from a clean login shell, and + note that `module avail` writes to stderr so grepping it needs `2>&1`. + + `slurm-batch` — for work that is per-sample rather than one command: + `sbatch` scripts, arrays throttled with `%N`, `--parsable` dependency + chains, and sizing the next run from `sacct`. Records the local numbers that + bite: `DefMemPerCPU` is 4000 MB so omitting `--mem` is not "unlimited", + `MaxArraySize` is 1001 so longer lists need chunking, `kill_invalid_depend` + is set so a dependent job vanishes rather than hangs when its upstream + fails, and `MaxRSS` is reported on the step rows where `sacct -X` will not + show it. + - A `slurm-discovery` skill, for finding out what the cluster actually offers instead of assuming it: what the partitions are and how big, which accounts and QOS the user holds, and the rule that decides whether a combination is diff --git a/Makefile b/Makefile index 4d49af6..27a8aee 100644 --- a/Makefile +++ b/Makefile @@ -54,15 +54,18 @@ install-system: # --------------------------------------------------------------------------- # Claude Code integration. Two parts: # -# - skills, which teach an agent how work is done here. bodhi-compute -# covers cluster etiquette: the login node and an sinteractive session are -# both orchestration shells, and real work goes into its own allocation. -# slurm-discovery covers finding out what the cluster offers — partitions, -# accounts, QOS and which combinations submit — and caching that map per -# cluster rather than rediscovering it. git-workflow covers the git -# conventions — semantic versioning, Conventional Commits, a worktree per -# branch, landing through a pull request — and is about the repository in -# the session, not the cluster; +# - skills, which teach an agent how work is done here, loaded on demand +# from their descriptions rather than all at once: +# +# bodhi-compute cluster etiquette; a session is not a compute target +# slurm-discovery partitions, accounts, QOS, and what you may submit +# bodhi-storage /beevol vs node-local /tmp, and where output belongs +# bodhi-software modules first, then containers, then pixi/uv +# slurm-batch sbatch, arrays, dependencies, sizing from sacct +# git-workflow semver, Conventional Commits, worktrees, PRs +# +# The first five are about the cluster; git-workflow is about the +# repository open in the session; # - two hooks for an agent running INSIDE a session, which tell it at # startup where it is and how big the allocation is, and warn it when the # session is running out of walltime. diff --git a/README.md b/README.md index f32325e..5349a1e 100644 --- a/README.md +++ b/README.md @@ -279,14 +279,20 @@ and new panes, but shells already running keep their original `SINTERACTIVE_NAME`. > [!TIP] -> This repo ships three [Claude Code skills](https://code.claude.com/docs/en/skills) -> plus two hooks, for agents that run **inside** a session. `bodhi-compute` -> teaches cluster etiquette; `slurm-discovery` teaches how to find out what -> the cluster actually offers you — partitions, accounts, QOS, and which -> combinations you may submit; `git-workflow` teaches the git conventions — -> semantic versioning, Conventional Commits, a worktree per branch, landing -> through a pull request. The hooks brief the agent on which session it is in -> at startup, and warn it when the session is running out of wall time. +> This repo ships six [Claude Code skills](https://code.claude.com/docs/en/skills) +> plus two hooks, for agents that run **inside** a session: +> +> | Skill | Teaches | +> |---|---| +> | `bodhi-compute` | Cluster etiquette — a session is not a compute target | +> | `slurm-discovery` | What the cluster offers: partitions, accounts, QOS, limits | +> | `bodhi-storage` | `/beevol` vs node-local `/tmp`, and where output belongs | +> | `bodhi-software` | Modules first, then containers, then pixi/uv | +> | `slurm-batch` | `sbatch`, arrays, dependencies, right-sizing from `sacct` | +> | `git-workflow` | Semver, Conventional Commits, worktrees, pull requests | +> +> The hooks brief the agent on which session it is in at startup, and warn it +> when the session is running out of wall time. > > ```bash > sinteractive --install-claude # from any installed copy diff --git a/docs/scripting.md b/docs/scripting.md index 50b1df5..97f44c3 100644 --- a/docs/scripting.md +++ b/docs/scripting.md @@ -148,8 +148,9 @@ need them too, since running `--install-claude` from inside a session runs the node's copy of the script. Point `SINTERACTIVE_SHARE` at a checkout to override, and `make nodes-check` to see which nodes actually have them. -**Three [skills](https://code.claude.com/docs/en/skills)** teach agents how -work is done here. +**Six [skills](https://code.claude.com/docs/en/skills)** teach agents how work +is done here. Skills load on demand from their descriptions, so an agent picks +up the one the task calls for rather than carrying all six. `bodhi-compute` covers cluster etiquette: neither the login node nor an sinteractive session is a compute target, real work goes into an allocation @@ -161,7 +162,26 @@ assuming it: what the partitions are and how big, which accounts and QOS you hold, and the rule that decides whether a given combination is submittable — your account in the partition's `AllowAccounts`, your QOS in both its `AllowQos` and your own association. It also covers reading `squeue`'s reason -column when a job is refused or sits `PENDING`. +column when a job is refused or sits `PENDING`, and caches the answers per +cluster so the survey is run once rather than every session. + +`bodhi-storage` covers where data goes: `/beevol` is one shared BeeGFS mount +and the compute node's `/tmp` is a local disk, so inputs are read from the +former and scratch is written to the latter and cleaned up on exit. It also +warns that `du` on a home directory can run for minutes, and that the shared +filesystem is full enough for a large write to be somebody else's problem too. + +`bodhi-software` covers how to get a tool: the module tree first — around 137 +preinstalled packages, so most of a genomics pipeline is a `module load` away +— then a container, then `pixi`/`uv` for the remainder. Pin the version, +load inside the job script rather than the login shell, and never `pip +install` into the system Python. + +`slurm-batch` covers work that is per-sample rather than a single command: +`sbatch` scripts, arrays and why to throttle them with `%N`, dependency +chains, and using `sacct` to size the next run from what the last one actually +used — noting that `MaxRSS` lives on the step rows, where `sacct -X` will not +show it. `git-workflow` covers the git conventions, and is about the repository open in the session rather than the cluster: semantic versioning with annotated diff --git a/man/sinteractive.1 b/man/sinteractive.1 index 5fa9efe..9e36571 100644 --- a/man/sinteractive.1 +++ b/man/sinteractive.1 @@ -486,12 +486,18 @@ run from inside one create their own allocations rather than steps of the session's job. .SS Claude Code hooks .B \-\-agent\-context -prints this briefing for the current session. Three skills and two hooks wire +prints this briefing for the current session. Six skills and two hooks wire it up \(em .I bodhi\-compute for cluster etiquette, .I slurm\-discovery -for mapping partitions, accounts and QOS, and +for partitions, accounts and QOS, +.I bodhi\-storage +for where data belongs, +.I bodhi\-software +for modules and environments, +.I slurm\-batch +for array jobs and dependencies, and .I git\-workflow for the git conventions, plus one hook briefing the agent at session start and one warning it when the session is running out of wall time \(em diff --git a/skills/bodhi-compute/SKILL.md b/skills/bodhi-compute/SKILL.md index a6fb9c9..fe13283 100644 --- a/skills/bodhi-compute/SKILL.md +++ b/skills/bodhi-compute/SKILL.md @@ -215,3 +215,10 @@ ssh NODE "${SINTERACTIVE_TMUX:-/usr/local/bin/tmux}" -L sinteractive-JOBID \ Cancel allocations you created as soon as the work is done — `scancel ID` for an `salloc`, `sinteractive --cancel JOBID|NAME` for a session. Never cancel a session you did not create without asking the user. + +## Related skills + +- `slurm-discovery` — which partitions, accounts and QOS you may actually use. +- `bodhi-storage` — read from `/beevol`, scratch on node-local `/tmp`. +- `bodhi-software` — check `module avail` before building or installing. +- `slurm-batch` — `sbatch`, arrays and dependencies, when it is not one job. diff --git a/skills/bodhi-software/SKILL.md b/skills/bodhi-software/SKILL.md new file mode 100644 index 0000000..d334711 --- /dev/null +++ b/skills/bodhi-software/SKILL.md @@ -0,0 +1,79 @@ +--- +name: bodhi-software +description: How software is provided on the Bodhi cluster — the module tree of ~137 preinstalled bioinformatics tools, Singularity containers, and user-level pixi/uv environments — and the order to try them in. Use before installing, building, or compiling anything, when a command is not found, or when choosing how to pin a tool version for a pipeline. +--- + +# Getting a tool + +**Look before you build.** The cluster already ships around 137 packages, and +building `bcftools` from source or pip-installing a bioinformatics tool that +is one `module load` away wastes an hour and produces a less reproducible +result. + +The order is: **module → container → pixi/uv**. + +## 1. Modules + +Environment Modules 5.3.0 (Tcl, not Lmod — there is no `module spider`): + +```bash +module avail # the whole catalogue +module avail 2>&1 | grep -i star # avail writes to STDERR; grep needs 2>&1 +module whatis samtools # what a name resolves to +module show samtools # what it puts on PATH, and its own deps +module load samtools/1.22.1 +module list +module purge # start clean +``` + +The tree at `/cluster/software/modules-sw` covers most of what a genomics +pipeline needs — aligners (`bwa`, `bowtie2`, `STAR`, `minimap2`, `hisat2`), +`samtools`/`bcftools`/`htslib`/`bedtools`/`bedops`, `cellranger` and friends, +`picard`, `ncbi-blast`, `salmon`, `kallisto`, `R` (4.3.3, 4.5.1, 4.5.2), +`java` (8 through 25), `plink`, `sratoolkit`. + +**Pin the version.** `module load STAR` takes whatever carries `(default)` +today, and defaults move — `module load STAR/2.7.11b` is what makes a run +reproducible next year. Modules also load their own dependencies (`samtools` +pulls in `htslib`), so do not hand-assemble a stack that `module show` already +describes. + +**Load inside the job, not just the login shell.** A module loaded in your +session is an environment change; `sbatch` starts from a fresh login shell and +will not have it. Put the `module load` lines in the job script. + +## 2. Containers + +```bash +singularity exec /path/to/image.sif command ... +singularity exec --bind /beevol:/beevol image.sif command ... +``` + +Reach for this when a tool is not in the module tree and comes with an +official image, or when a pipeline pins one. Bind `/beevol` explicitly if the +tool needs to see cluster paths. + +## 3. pixi and uv + +For everything left over, and for project-local environments that belong to a +repository rather than to the cluster: + +```bash +pixi add samtools # project env, recorded in pixi.toml +pixi global install jq # a small tool you want on PATH everywhere +uv venv && uv pip install ... # Python projects +``` + +Both live under `$HOME`, which is shared across every node, so an environment +built once in a session works in every allocation without reinstalling. + +**Never `pip install` into the system Python.** It is not writable, and +`--user` puts packages on a path every job inherits, which turns one project's +pin into every project's problem. Use a project environment. + +## Building from source + +Only after checking `module avail`. If you do build, it is real compute — +give it its own allocation rather than running it in the session shell (see +`bodhi-compute`), install into `$HOME` or a project prefix, and write down in +the project what was built and why the module tree was not enough. diff --git a/skills/bodhi-storage/SKILL.md b/skills/bodhi-storage/SKILL.md new file mode 100644 index 0000000..fc9e7b8 --- /dev/null +++ b/skills/bodhi-storage/SKILL.md @@ -0,0 +1,86 @@ +--- +name: bodhi-storage +description: Where data lives on the Bodhi cluster and where to write it — the shared /beevol BeeGFS filesystem versus node-local /tmp, which one intermediates belong on, and how to check space without hanging the session. Use before writing large output, staging inputs, choosing a working directory for a job, or when a job is slow at I/O or the filesystem is full. +--- + +# Where does it go? + +## One shared filesystem, and one local disk + +```bash +df -h /beevol /tmp +# beegfs_nodev 839T 700T 139T 84% /beevol <- shared, everyone +# /dev/mapper/system-root 423G 22G 401G 6% / <- this node only +``` + +`/beevol` is a single BeeGFS mount and the only thing shared between nodes: + +| Path | What it is | +|---|---| +| `/beevol/home/$USER` | Your home. Code, environments, results worth keeping. | +| `/beevol/data` | Shared reference and project data. | +| `/beevol/illumina` | Sequencer output — `runs/`, `data/`. Read from it, don't write to it. | + +`/tmp` is the compute node's own 423G disk (Slurm's `TmpFS`), and `/dev/shm` +is a 377G tmpfs — RAM, so anything you put there counts against your job's +`--mem` and disappears with the allocation. + +**The filesystem is 84% full and it is shared with everyone.** Space you free +is space someone else's run does not fail for. + +## The rule + +**Read inputs from `/beevol`, write scratch to node-local `/tmp`, copy the +results back.** A pipeline that streams thousands of small writes to BeeGFS +is slow for you and slow for everyone else on the cluster; the same work +against local disk is not. + +Slurm does not hand out a private temp directory here — `TMPDIR` is plain +`/tmp` and `SLURM_TMPDIR` is unset — so make your own and clean it up, +because nothing else will: + +```bash +#!/usr/bin/env bash +set -euo pipefail +work=/tmp/$USER-$SLURM_JOB_ID +mkdir -p "$work" +trap 'rm -rf "$work"' EXIT # runs on success, failure, and scancel + +samtools sort -@ 8 -T "$work"/sort -o "$work"/out.bam /beevol/data/in.bam +cp "$work"/out.bam /beevol/home/$USER/results/ +``` + +The `trap` matters: `/tmp` on a shared node already has a couple of thousand +entries, and an uncleaned job directory sits there until someone notices. + +Keep the *final* artifacts on `/beevol` — `/tmp` is node-local, so the next +job in the pipeline probably lands somewhere else and cannot see it. + +## Checking space + +`df` is instant. **`du` on a home directory is not** — BeeGFS has to walk +every file, and `du -sh /beevol/home/$USER` can run for many minutes and give +you nothing to show for it. Point it at a subdirectory you actually suspect, +give it a timeout, and run it in an allocation rather than in the session: + +```bash +df -h /beevol # always safe +du -sh --max-depth=1 ~/devel 2>/dev/null # one level, one subtree +``` + +For finding what to delete, target the big and the old rather than +summarising everything: + +```bash +find ~/ -xdev -type f -size +5G -printf '%s\t%p\n' 2>/dev/null | sort -rn | head +``` + +## Before writing something large + +Estimate the output, check `df -h /beevol`, and say so if the run would take a +visible bite out of the remaining 139T. Ask the user before writing hundreds +of gigabytes to shared storage — on a filesystem this full that is a decision +about other people's work, not just theirs. + +Sizing an allocation for the job that does the writing is the `bodhi-compute` +skill; finding out which partition you may use is `slurm-discovery`. diff --git a/skills/slurm-batch/SKILL.md b/skills/slurm-batch/SKILL.md new file mode 100644 index 0000000..d7a3d92 --- /dev/null +++ b/skills/slurm-batch/SKILL.md @@ -0,0 +1,116 @@ +--- +name: slurm-batch +description: Running many jobs with sbatch — job scripts, array jobs and their throttling, dependencies between stages, and using sacct to right-size the next run from what the last one actually used. Use when the work is per-sample or per-file rather than a single command, when building a multi-stage pipeline, or when sizing memory and walltime for a batch. +--- + +# Many jobs, not one + +`srun` and `salloc` (the `bodhi-compute` skill) are for one thing at a time, +attached. A batch of hundreds of samples is a different shape: it is submitted +and left, and its sizing is decided once and then applied hundreds of times — +which is what makes getting the sizing right worth a few minutes up front. + +## A job script + +```bash +#!/usr/bin/env bash +#SBATCH --job-name=star-align +#SBATCH --comment=star-align +#SBATCH --partition=rna +#SBATCH --account=rbi +#SBATCH --cpus-per-task=8 +#SBATCH --mem=32G +#SBATCH --time=4:00:00 +#SBATCH --output=logs/%x-%j.out +#SBATCH --error=logs/%x-%j.err +set -euo pipefail + +module load STAR/2.7.11b # sbatch starts from a clean login shell +... +``` + +Name it in **both** `--job-name` and `--comment`, the same short descriptive +value, for the same reason as `srun` — a shared queue should say what is +running and why. + +`--output` directories are not created for you: `mkdir -p logs` first, or the +job dies at startup with nowhere to write. + +**`DefMemPerCPU` is 4000 MB here.** Leaving `--mem` off does not mean +"unlimited", it means 4G per CPU — a frequent and confusing cause of a job +being killed for memory it never asked for. + +## Arrays + +One submission, one task per sample: + +```bash +#SBATCH --array=0-499%20 # 500 tasks, at most 20 running at once +sample=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" samples.txt) +``` + +- **`MaxArraySize` is 1001** on this cluster, so indices run `0-1000` and a + list longer than that has to be chunked into several submissions. +- **Throttle with `%N`.** Without it, 500 tasks all become eligible at once, + which fills the partition and pushes everyone else — including your own + later stages — behind them. `%20` is a courteous default; raise it when the + partition is idle. +- `%A` is the array job id and `%a` the task id, so + `--output=logs/%x-%A_%a.out` keeps per-task logs apart. `%j` alone collides. +- `MaxJobCount` is 10000 cluster-wide, and the QOS caps submissions per user + (`normal` allows 2000 submitted, 500 running). `slurm-discovery` covers + reading those. + +## Dependencies + +```bash +a=$(sbatch --parsable align.sh) +b=$(sbatch --parsable --dependency=afterok:$a merge.sh) +sbatch --dependency=afterok:$b report.sh +``` + +`--parsable` prints the bare job id, which is what makes this chainable. +`afterok` waits for success; `afterany` runs regardless; `singleton` serialises +jobs sharing a name. + +This cluster runs with `kill_invalid_depend`, so a dependency that can never +be satisfied — the job it waits on failed — is **killed rather than left +pending forever**. A vanished downstream job usually means an upstream +failure, so check that first with `sacct` rather than resubmitting. + +## Right-size from what actually happened + +Run one sample, then look at what it used before committing to hundreds: + +```bash +sacct -j JOBID --format=JobID,JobName%14,ReqMem,MaxRSS,AllocCPUS,Elapsed,State +``` + +**`MaxRSS` is reported on the step rows, not the allocation row.** The parent +line is blank and `sacct -X`, which shows allocations only, hides it entirely: + +``` +237176 bash 8G 4 00:00:36 COMPLETED +237176.0 bash 631852K 4 00:00:36 COMPLETED <- here +``` + +That job asked for 8G and touched 617M. Ask for what the measurement says plus +headroom, not a round number that felt safe — over-requesting memory and +walltime is what makes a queue slow for everybody, since the scheduler must +find a hole big enough for the request rather than the job. + +Walltime is the same trade in reverse: too short and the job is killed at the +limit with its output half-written, too long and backfill will not slot it in. +`Elapsed` from the trial run is the number to build on. + +## Watching and cleaning up + +```bash +squeue --me -o "%.10i %.20j %.10T %.10M %r" # %r is the reason if pending +scancel JOBID # whole array +scancel JOBID_7 # one task +scancel --me --name=star-align # by name +``` + +Cancel a batch you have abandoned rather than leaving it to drain the +partition. Never cancel jobs you did not submit.