From 2ecb3197ee5a4f20f52e6c27ef0ec1913dc64d42 Mon Sep 17 00:00:00 2001 From: Jay Hesselberth Date: Wed, 26 Aug 2026 08:12:53 -0600 Subject: [PATCH] docs(claude): teach bodhi-compute to size walltime around maintenance windows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bodhi takes a maintenance reservation about once a month and it carries ALL_NODES, so there is nowhere on the cluster to run during the window. The failure mode is quiet, which is what makes it worth writing down: a job asking for more walltime than remains before the start is not rejected, it is deferred to after the window. The job queues, looks entirely normal, and the cost shows up only if someone checks. Measured against a window 21h49m out, -t 21:00:00 started immediately and -t 22:00:00 was pushed to the reservation's end time. One extra hour of requested walltime bought two days of waiting. The section covers reading scontrol show reservation, computing the gap, confirming with srun --test-only, and recognising the aftermath in squeue. Two real jobs were sitting in exactly that state while this was written, both having asked for three days with the window a day out: 243393 krausmeg 3-00:00:00 ReqNodeNotAvail, Reserved for maintenance That state is not stuck and does not want resubmitting; shortening -t is what makes it run sooner. IGNORE_JOBS means jobs already running when the reservation was created are not killed up front, but nothing survives the window itself — sinteractive sessions included, so a session should be launched to end before the start rather than reach past it. slurm-discovery gains the matching reason row, and a note that reservations are weather rather than structure: they recur monthly but each one carries a date, so they are read live and never written to the cached map, where a stale window is exactly the sort of thing that would get trusted. Every command, boundary and output here was measured against Bodhi. The gap snippet was extracted back out of the finished skill and run verbatim. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 19 +++++++++ skills/bodhi-compute/SKILL.md | 69 +++++++++++++++++++++++++++++++++ skills/slurm-discovery/SKILL.md | 7 ++++ 3 files changed, 95 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3434ec3..5029bff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,25 @@ and this project adheres to ### Added +- `bodhi-compute` now covers Bodhi's monthly maintenance reservation, and + sizing walltime around it. The reservation carries `ALL_NODES`, so there is + nowhere on the cluster to run during the window, and the failure mode is + quiet: a job asking for more walltime than remains before the start is not + rejected, it is **deferred to after the window**. Measured against a window + 21h49m out, `-t 21:00:00` started immediately and `-t 22:00:00` was pushed + to the reservation's end time — one extra hour of request bought two days of + waiting. The section covers reading `scontrol show reservation`, computing + the gap, confirming with `srun --test-only`, and recognising + `ReqNodeNotAvail, Reserved for maintenance` in `squeue` as "waiting for the + cluster to come back", not something to resubmit. `IGNORE_JOBS` means jobs + already running are not killed when the reservation is created, but nothing + survives the window itself — sinteractive sessions included, so a session + should be launched to end before the start rather than reach past it. + + `slurm-discovery` gains the matching `squeue` reason, and a note that + reservations are weather rather than structure: they recur monthly but each + one has a date, so they are read live and never written to the cached map. + - Three more skills covering the things an agent hits in the first ten minutes of real work on the cluster: diff --git a/skills/bodhi-compute/SKILL.md b/skills/bodhi-compute/SKILL.md index fe13283..2f9b940 100644 --- a/skills/bodhi-compute/SKILL.md +++ b/skills/bodhi-compute/SKILL.md @@ -106,6 +106,75 @@ them, so the right `-p` can still be rejected under the wrong `-A`. The `slurm-discovery` skill covers mapping that out, and reading the reason when a job is refused or stuck `PENDING`. +### Check for a maintenance window before asking for walltime + +Bodhi takes a maintenance reservation roughly once a month, and it covers +**every node on the cluster**. Check before requesting anything long: + +```bash +scontrol show reservation +# ReservationName=monthly-maint StartTime=2026-08-27T06:00:00 +# EndTime=2026-08-28T06:00:00 Duration=1-00:00:00 +# Nodes=compgpu[01-03],compute[00-21] NodeCnt=25 +# Flags=MAINT,IGNORE_JOBS,SPEC_NODES,ALL_NODES +# Users=root State=INACTIVE +``` + +`State=INACTIVE` means it has not started yet; `ACTIVE` means it is on and the +nodes are gone. `Users=root` means it is not a reservation you can submit +into. `ALL_NODES` is the part that matters — there is nowhere else to go. + +**A job asking for more walltime than remains before the window does not +fail. It gets deferred to after the window.** That is the whole hazard: the +job queues and looks normal, and the cost is invisible unless you check. With +the window 21h49m away: + +```console +$ srun --test-only -p rna -c 4 --mem 8G -t 21:00:00 -- true +srun: Job 245072 to start at 2026-08-26T08:10:20 ... # starts now + +$ srun --test-only -p rna -c 4 --mem 8G -t 22:00:00 -- true +srun: Job 245073 to start at 2026-08-28T06:00:00 ... # +46 hours +``` + +One extra hour of requested walltime cost two days of waiting. So **size `-t` +to fit in the gap**, and if the work genuinely cannot fit, say so and let the +user decide between splitting it and waiting. + +How long is the gap: + +```bash +scontrol show reservation 2>/dev/null | + sed -n 's/.*ReservationName=\([^ ]*\) StartTime=\([^ ]*\) EndTime=\([^ ]*\).*/\1 \2 \3/p' | + while read -r name start end; do + s=$(date -d "$start" +%s) now=$(date +%s) + ((s > now)) && printf '%s starts in %dh%02dm\n' \ + "$name" $(((s - now) / 3600)) $((((s - now) % 3600) / 60)) + done +# monthly-maint starts in 21h49m +``` + +`--test-only` is the cheap confirmation either way — it reports the start time +the scheduler would actually give the job without queueing anything. + +After the fact, a job caught this way says so in `squeue`: + +```console +$ squeue -t PENDING -o "%.10i %.12u %.12L %r" + 243393 krausmeg 3-00:00:00 ReqNodeNotAvail, Reserved for maintenance +``` + +Both of those asked for three days with the window a day out. A job in that +state is not stuck and does not need resubmitting — it is waiting for the +cluster to come back. Shortening `-t` is what makes it run sooner. + +`IGNORE_JOBS` means the reservation was allowed to be created over jobs that +were already running, so those are not killed up front — but they do not +survive the window either. **Nothing running is safe across it**, sinteractive +sessions included: a session whose walltime crosses the start time will be cut +short, so launch one that ends before the window rather than one that reaches +past it. + ## sinteractive sessions These are the user's persistent interactive shells. You mostly *observe* them; diff --git a/skills/slurm-discovery/SKILL.md b/skills/slurm-discovery/SKILL.md index 97a13e5..5363b88 100644 --- a/skills/slurm-discovery/SKILL.md +++ b/skills/slurm-discovery/SKILL.md @@ -58,6 +58,12 @@ below keeps: what belongs in the file is the structure — which partitions exist, how big they are, what you may ask for, and the limits on it. Anything about right now gets run live, every time. +Maintenance reservations are the tempting exception, and they are weather too: +they recur monthly but each one has a date, and a stale window in a file is +exactly the sort of thing that gets trusted. `scontrol show reservation` is +one call — run it, do not cache it. See `bodhi-compute` for what to do with +the answer. + ## The survey **What partitions exist, and how big are they?** @@ -159,6 +165,7 @@ The reason names the wall you hit: | `QOSMaxCpuPerUserLimit`, `AssocMaxJobsLimit` | Your own running jobs are holding the budget. | | `PartitionTimeLimit`, `PartitionConfig` | The request cannot fit the partition at all. | | `ReqNodeNotAvail` | Named nodes are down or drained — check `sinfo -R`. | +| `ReqNodeNotAvail, Reserved for maintenance` | `-t` reaches past a maintenance window, so the job is deferred until after it. Shorten it — see `bodhi-compute`. | For a job already rejected at submit, re-run with `--test-only` to get the verdict without queueing anything: