feat(claude): add slurm-discovery, bodhi-storage, bodhi-software and slurm-batch skills

CHANGELOG.md

@@ -10,6 +10,51 @@ and this project adheres to
 
 ### Added
 
+- Three more skills covering the things an agent hits in the first ten minutes
+  of real work on the cluster:
+
+  `bodhi-storage` — `/beevol` is one shared BeeGFS mount and the compute
+  node's `/tmp` is a 423G local disk, so inputs are read from the former and
+  scratch written to the latter and cleaned up with a `trap`. Slurm hands out
+  no private temp directory here (`TMPDIR` is plain `/tmp`, `SLURM_TMPDIR` is
+  unset), which is why uncleaned job directories accumulate. It also records
+  that `du` on a home directory can run for minutes on BeeGFS, and that at 84%
+  full a large write is somebody else's problem too.
+
+  `bodhi-software` — the order is module, then container, then `pixi`/`uv`.
+  The tree at `/cluster/software/modules-sw` carries around 137 packages, so
+  most of a genomics pipeline is a `module load` away and building it from
+  source is wasted time. Pin the version rather than taking `(default)`, load
+  inside the job script because `sbatch` starts from a clean login shell, and
+  note that `module avail` writes to stderr so grepping it needs `2>&1`.
+
+  `slurm-batch` — for work that is per-sample rather than one command:
+  `sbatch` scripts, arrays throttled with `%N`, `--parsable` dependency
+  chains, and sizing the next run from `sacct`. Records the local numbers that
+  bite: `DefMemPerCPU` is 4000 MB so omitting `--mem` is not "unlimited",
+  `MaxArraySize` is 1001 so longer lists need chunking, `kill_invalid_depend`
+  is set so a dependent job vanishes rather than hangs when its upstream
+  fails, and `MaxRSS` is reported on the step rows where `sacct -X` will not
+  show it.
+
+- A `slurm-discovery` skill, for finding out what the cluster actually offers
+  instead of assuming it: what the partitions are and how big, which accounts
+  and QOS the user holds, and the rule that decides whether a combination is
+  submittable — your account in the partition's `AllowAccounts`, and the QOS
+  you ask for in both its `AllowQos` and your own association. That
+  intersection is the part nobody guesses right: on Bodhi the `gpu` partition
+  takes `gpu_rbi`/`gpu_devbio`/`gpu_scb` and not the default `rbi` account, so
+  the request is refused however many GPUs are idle, and the error names
+  neither half. It also covers reading the QOS limit columns, and `squeue`'s
+  reason column when a job is rejected or sits `PENDING`.
+
+  The survey's answers are cached to
+  `~/.cache/sinteractive/slurm-map-<cluster>.md` and re-read rather than
+  re-run. Keyed by `ClusterName` because one `$HOME` is often mounted on
+  several clusters, and a map from the wrong one is worse than none. Only the
+  structure is cached — node states and queue depth are re-read live every
+  time, so the cached `sinfo` deliberately drops the state column.
+
 - A second Claude Code skill, `git-workflow`, installed alongside
   `bodhi-compute` by `--install-claude`. Where `bodhi-compute` is about the
   cluster, this one is about the repository open in the session: semantic

Makefile

@@ -54,12 +54,18 @@ install-system:
 # ---------------------------------------------------------------------------
 # Claude Code integration. Two parts:
 #
-#   - skills, which teach an agent how work is done here. bodhi-compute
-#     covers cluster etiquette: the login node and an sinteractive session are
-#     both orchestration shells, and real work goes into its own allocation.
-#     git-workflow covers the git conventions — semantic versioning,
-#     Conventional Commits, a worktree per branch, landing through a pull
-#     request — and is about the repository in the session, not the cluster;
+#   - skills, which teach an agent how work is done here, loaded on demand
+#     from their descriptions rather than all at once:
+#
+#       bodhi-compute    cluster etiquette; a session is not a compute target
+#       slurm-discovery  partitions, accounts, QOS, and what you may submit
+#       bodhi-storage    /beevol vs node-local /tmp, and where output belongs
+#       bodhi-software   modules first, then containers, then pixi/uv
+#       slurm-batch      sbatch, arrays, dependencies, sizing from sacct
+#       git-workflow     semver, Conventional Commits, worktrees, PRs
+#
+#     The first five are about the cluster; git-workflow is about the
+#     repository open in the session;
 #   - two hooks for an agent running INSIDE a session, which tell it at
 #     startup where it is and how big the allocation is, and warn it when the
 #     session is running out of walltime.

README.md

@@ -279,12 +279,20 @@ and new panes, but shells already running keep their original
 `SINTERACTIVE_NAME`.
 
 > [!TIP]
-> This repo ships two [Claude Code skills](https://code.claude.com/docs/en/skills)
-> plus two hooks, for agents that run **inside** a session. `bodhi-compute`
-> teaches cluster etiquette; `git-workflow` teaches the git conventions —
-> semantic versioning, Conventional Commits, a worktree per branch, landing
-> through a pull request. The hooks brief the agent on which session it is in
-> at startup, and warn it when the session is running out of wall time.
+> This repo ships six [Claude Code skills](https://code.claude.com/docs/en/skills)
+> plus two hooks, for agents that run **inside** a session:
+>
+> | Skill | Teaches |
+> |---|---|
+> | `bodhi-compute` | Cluster etiquette — a session is not a compute target |
+> | `slurm-discovery` | What the cluster offers: partitions, accounts, QOS, limits |
+> | `bodhi-storage` | `/beevol` vs node-local `/tmp`, and where output belongs |
+> | `bodhi-software` | Modules first, then containers, then pixi/uv |
+> | `slurm-batch` | `sbatch`, arrays, dependencies, right-sizing from `sacct` |
+> | `git-workflow` | Semver, Conventional Commits, worktrees, pull requests |
+>
+> The hooks brief the agent on which session it is in at startup, and warn it
+> when the session is running out of wall time.
 >
 > ```bash
 > sinteractive --install-claude   # from any installed copy

docs/scripting.md

@@ -148,14 +148,41 @@ need them too, since running `--install-claude` from inside a session runs
 the node's copy of the script. Point `SINTERACTIVE_SHARE` at a checkout to
 override, and `make nodes-check` to see which nodes actually have them.
 
-**Two [skills](https://code.claude.com/docs/en/skills)** teach agents how work
-is done here.
+**Six [skills](https://code.claude.com/docs/en/skills)** teach agents how work
+is done here. Skills load on demand from their descriptions, so an agent picks
+up the one the task calls for rather than carrying all six.
 
 `bodhi-compute` covers cluster etiquette: neither the login node nor an
 sinteractive session is a compute target, real work goes into an allocation
 sized for it, reuse sessions rather than piling them up, check the time budget
 before long jobs, and clean up.
 
+`slurm-discovery` covers finding out what the cluster offers rather than
+assuming it: what the partitions are and how big, which accounts and QOS you
+hold, and the rule that decides whether a given combination is submittable —
+your account in the partition's `AllowAccounts`, your QOS in both its
+`AllowQos` and your own association. It also covers reading `squeue`'s reason
+column when a job is refused or sits `PENDING`, and caches the answers per
+cluster so the survey is run once rather than every session.
+
+`bodhi-storage` covers where data goes: `/beevol` is one shared BeeGFS mount
+and the compute node's `/tmp` is a local disk, so inputs are read from the
+former and scratch is written to the latter and cleaned up on exit. It also
+warns that `du` on a home directory can run for minutes, and that the shared
+filesystem is full enough for a large write to be somebody else's problem too.
+
+`bodhi-software` covers how to get a tool: the module tree first — around 137
+preinstalled packages, so most of a genomics pipeline is a `module load` away
+— then a container, then `pixi`/`uv` for the remainder. Pin the version,
+load inside the job script rather than the login shell, and never `pip
+install` into the system Python.
+
+`slurm-batch` covers work that is per-sample rather than a single command:
+`sbatch` scripts, arrays and why to throttle them with `%N`, dependency
+chains, and using `sacct` to size the next run from what the last one actually
+used — noting that `MaxRSS` lives on the step rows, where `sacct -X` will not
+show it.
+
 `git-workflow` covers the git conventions, and is about the repository open in
 the session rather than the cluster: semantic versioning with annotated
 `vX.Y.Z` tags, Conventional Commit messages, one worktree per branch under

man/sinteractive.1

@@ -486,10 +486,18 @@ run from inside one create their own allocations rather than steps of the
 session's job.
 .SS Claude Code hooks
 .B \-\-agent\-context
-prints this briefing for the current session. Two skills and two hooks wire
+prints this briefing for the current session. Six skills and two hooks wire
 it up \(em
 .I bodhi\-compute
-for cluster etiquette and
+for cluster etiquette,
+.I slurm\-discovery
+for partitions, accounts and QOS,
+.I bodhi\-storage
+for where data belongs,
+.I bodhi\-software
+for modules and environments,
+.I slurm\-batch
+for array jobs and dependencies, and
 .I git\-workflow
 for the git conventions, plus one hook briefing the agent at session start
 and one warning it when the session is running out of wall time \(em

skills/bodhi-compute/SKILL.md

@@ -101,6 +101,11 @@ for sinteractive sessions and is the smallest partition on the cluster
 Request only what the task needs, and ask the user before requesting more than
 a day of walltime or a whole node's worth of resources.
 
+`gpu` and some other partitions restrict which accounts and QOS may submit to
+them, so the right `-p` can still be rejected under the wrong `-A`. The
+`slurm-discovery` skill covers mapping that out, and reading the reason when a
+job is refused or stuck `PENDING`.
+
 ## sinteractive sessions
 
 These are the user's persistent interactive shells. You mostly *observe* them;
@@ -210,3 +215,10 @@ ssh NODE "${SINTERACTIVE_TMUX:-/usr/local/bin/tmux}" -L sinteractive-JOBID \
 Cancel allocations you created as soon as the work is done — `scancel ID` for
 an `salloc`, `sinteractive --cancel JOBID|NAME` for a session. Never cancel a
 session you did not create without asking the user.
+
+## Related skills
+
+- `slurm-discovery` — which partitions, accounts and QOS you may actually use.
+- `bodhi-storage` — read from `/beevol`, scratch on node-local `/tmp`.
+- `bodhi-software` — check `module avail` before building or installing.
+- `slurm-batch` — `sbatch`, arrays and dependencies, when it is not one job.

skills/bodhi-software/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: bodhi-software
+description: How software is provided on the Bodhi cluster — the module tree of ~137 preinstalled bioinformatics tools, Singularity containers, and user-level pixi/uv environments — and the order to try them in. Use before installing, building, or compiling anything, when a command is not found, or when choosing how to pin a tool version for a pipeline.
+---
+
+# Getting a tool
+
+**Look before you build.** The cluster already ships around 137 packages, and
+building `bcftools` from source or pip-installing a bioinformatics tool that
+is one `module load` away wastes an hour and produces a less reproducible
+result.
+
+The order is: **module → container → pixi/uv**.
+
+## 1. Modules
+
+Environment Modules 5.3.0 (Tcl, not Lmod — there is no `module spider`):
+
+```bash
+module avail                          # the whole catalogue
+module avail 2>&1 | grep -i star      # avail writes to STDERR; grep needs 2>&1
+module whatis samtools                # what a name resolves to
+module show samtools                  # what it puts on PATH, and its own deps
+module load samtools/1.22.1
+module list
+module purge                          # start clean
+```
+
+The tree at `/cluster/software/modules-sw` covers most of what a genomics
+pipeline needs — aligners (`bwa`, `bowtie2`, `STAR`, `minimap2`, `hisat2`),
+`samtools`/`bcftools`/`htslib`/`bedtools`/`bedops`, `cellranger` and friends,
+`picard`, `ncbi-blast`, `salmon`, `kallisto`, `R` (4.3.3, 4.5.1, 4.5.2),
+`java` (8 through 25), `plink`, `sratoolkit`.
+
+**Pin the version.** `module load STAR` takes whatever carries `(default)`
+today, and defaults move — `module load STAR/2.7.11b` is what makes a run
+reproducible next year. Modules also load their own dependencies (`samtools`
+pulls in `htslib`), so do not hand-assemble a stack that `module show` already
+describes.
+
+**Load inside the job, not just the login shell.** A module loaded in your
+session is an environment change; `sbatch` starts from a fresh login shell and
+will not have it. Put the `module load` lines in the job script.
+
+## 2. Containers
+
+```bash
+singularity exec /path/to/image.sif command ...
+singularity exec --bind /beevol:/beevol image.sif command ...
+```
+
+Reach for this when a tool is not in the module tree and comes with an
+official image, or when a pipeline pins one. Bind `/beevol` explicitly if the
+tool needs to see cluster paths.
+
+## 3. pixi and uv
+
+For everything left over, and for project-local environments that belong to a
+repository rather than to the cluster:
+
+```bash
+pixi add samtools                    # project env, recorded in pixi.toml
+pixi global install jq               # a small tool you want on PATH everywhere
+uv venv && uv pip install ...        # Python projects
+```
+
+Both live under `$HOME`, which is shared across every node, so an environment
+built once in a session works in every allocation without reinstalling.
+
+**Never `pip install` into the system Python.** It is not writable, and
+`--user` puts packages on a path every job inherits, which turns one project's
+pin into every project's problem. Use a project environment.
+
+## Building from source
+
+Only after checking `module avail`. If you do build, it is real compute —
+give it its own allocation rather than running it in the session shell (see
+`bodhi-compute`), install into `$HOME` or a project prefix, and write down in
+the project what was built and why the module tree was not enough.

skills/bodhi-storage/SKILL.md

@@ -0,0 +1,86 @@
+---
+name: bodhi-storage
+description: Where data lives on the Bodhi cluster and where to write it — the shared /beevol BeeGFS filesystem versus node-local /tmp, which one intermediates belong on, and how to check space without hanging the session. Use before writing large output, staging inputs, choosing a working directory for a job, or when a job is slow at I/O or the filesystem is full.
+---
+
+# Where does it go?
+
+## One shared filesystem, and one local disk
+
+```bash
+df -h /beevol /tmp
+# beegfs_nodev              839T  700T  139T  84%  /beevol      <- shared, everyone
+# /dev/mapper/system-root   423G   22G  401G   6%  /            <- this node only
+```
+
+`/beevol` is a single BeeGFS mount and the only thing shared between nodes:
+
+| Path | What it is |
+|---|---|
+| `/beevol/home/$USER` | Your home. Code, environments, results worth keeping. |
+| `/beevol/data` | Shared reference and project data. |
+| `/beevol/illumina` | Sequencer output — `runs/`, `data/`. Read from it, don't write to it. |
+
+`/tmp` is the compute node's own 423G disk (Slurm's `TmpFS`), and `/dev/shm`
+is a 377G tmpfs — RAM, so anything you put there counts against your job's
+`--mem` and disappears with the allocation.
+
+**The filesystem is 84% full and it is shared with everyone.** Space you free
+is space someone else's run does not fail for.
+
+## The rule
+
+**Read inputs from `/beevol`, write scratch to node-local `/tmp`, copy the
+results back.** A pipeline that streams thousands of small writes to BeeGFS
+is slow for you and slow for everyone else on the cluster; the same work
+against local disk is not.
+
+Slurm does not hand out a private temp directory here — `TMPDIR` is plain
+`/tmp` and `SLURM_TMPDIR` is unset — so make your own and clean it up,
+because nothing else will:
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+work=/tmp/$USER-$SLURM_JOB_ID
+mkdir -p "$work"
+trap 'rm -rf "$work"' EXIT          # runs on success, failure, and scancel
+
+samtools sort -@ 8 -T "$work"/sort -o "$work"/out.bam /beevol/data/in.bam
+cp "$work"/out.bam /beevol/home/$USER/results/
+```
+
+The `trap` matters: `/tmp` on a shared node already has a couple of thousand
+entries, and an uncleaned job directory sits there until someone notices.
+
+Keep the *final* artifacts on `/beevol` — `/tmp` is node-local, so the next
+job in the pipeline probably lands somewhere else and cannot see it.
+
+## Checking space
+
+`df` is instant. **`du` on a home directory is not** — BeeGFS has to walk
+every file, and `du -sh /beevol/home/$USER` can run for many minutes and give
+you nothing to show for it. Point it at a subdirectory you actually suspect,
+give it a timeout, and run it in an allocation rather than in the session:
+
+```bash
+df -h /beevol                                   # always safe
+du -sh --max-depth=1 ~/devel 2>/dev/null        # one level, one subtree
+```
+
+For finding what to delete, target the big and the old rather than
+summarising everything:
+
+```bash
+find ~/ -xdev -type f -size +5G -printf '%s\t%p\n' 2>/dev/null | sort -rn | head
+```
+
+## Before writing something large
+
+Estimate the output, check `df -h /beevol`, and say so if the run would take a
+visible bite out of the remaining 139T. Ask the user before writing hundreds
+of gigabytes to shared storage — on a filesystem this full that is a decision
+about other people's work, not just theirs.
+
+Sizing an allocation for the job that does the writing is the `bodhi-compute`
+skill; finding out which partition you may use is `slurm-discovery`.