feat(claude): add slurm-discovery, bodhi-storage, bodhi-software and slurm-batch skills

[jayhesselberth]

Four skills covering what an agent needs before it can do real work on the
cluster: which partition it may use, where the output goes, where the tool
comes from, and how to run the same command four hundred times. Every command
and number in them was checked against Bodhi rather than recalled.

No installer change is needed for any of it — skills are discovered from what
ships beside the script, so each one is a new directory.

slurm-discovery

bodhi-compute tells an agent to pick a partition with sinfo, which is half
the story. Partitions restrict which accounts and QOS may submit, and a
request must satisfy both the partition's AllowAccounts/AllowQos and
the user's own association. Neither half implies the other, and failing either
gives the same unhelpful "invalid account or partition".

That intersection is the part nobody guesses right. On Bodhi the gpu
partition takes gpu_rbi/gpu_devbio/gpu_scb, not the default rbi, so
the job is refused however many GPUs are idle — and an agent that does not
know to look ends up shrinking a request that was never too big.

Covers the survey, the rule, what MaxWall/MaxTRESPU/OverPartQOS/
DenyOnLimit do to a request, and a table mapping squeue's reason column to
the wall you hit, plus srun --test-only.

The map is cached to ~/.cache/sinteractive/slurm-map-<cluster>.md and
re-read rather than re-run, since partitions and accounts change on the order
of months. Keyed by ClusterName, because one $HOME is often mounted on
several clusters and a map from the wrong one is worse than none. Only the
structure is cached — the cached sinfo deliberately drops the state column,
since a cached idle is a lie within minutes.

bodhi-storage

/beevol is one shared BeeGFS mount at 84% full; the compute node's
/tmp is a 423G local disk. Read inputs from the former, write scratch to the
latter, copy results back — a pipeline streaming small writes to BeeGFS is
slow for its owner and slow for everyone else.

Slurm hands out no private temp directory here (TMPDIR is plain /tmp,
SLURM_TMPDIR unset), so the skill shows making one under the job id and
clearing it with a trap — which is why /tmp on a shared node is already
two thousand entries deep. It also records that du on a home directory can
run for minutes on BeeGFS, long enough to look like a hang.

bodhi-software

Module → container → pixi/uv. /cluster/software/modules-sw carries ~137
packages, so most of a genomics pipeline is one module load away and
building from source is an hour spent on a less reproducible result. Pin the
version rather than taking today's (default); load inside the job script,
since sbatch starts from a clean login shell; and note module avail writes
to stderr, so grepping it without 2>&1 silently returns nothing.

slurm-batch

For work that is per-sample rather than one command: job scripts, arrays
throttled with %N, --parsable dependency chains, and sizing the next run
from what the last one measured. The local numbers that bite are written down
rather than left to be discovered:

• DefMemPerCPU is 4000 MB — omitting --mem is not "unlimited"
• MaxArraySize is 1001 — longer sample lists need chunking
• kill_invalid_depend is set — a dependent job vanishes rather than hangs
  when its upstream fails
• MaxRSS is on the step rows, where sacct -X will not show it

Also

bodhi-compute gains a related-skills list; the README, docs/scripting.md,
man page and Makefile comment now describe six skills. All four
validate.yml gates pass locally, --install-claude into a throwaway config
dir picks up all six, and every skill's frontmatter was checked for a
name: matching its directory.

🤖 Generated with Claude Code