Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,25 @@ and this project adheres to

### Added

- `bodhi-compute` now covers Bodhi's monthly maintenance reservation, and
sizing walltime around it. The reservation carries `ALL_NODES`, so there is
nowhere on the cluster to run during the window, and the failure mode is
quiet: a job asking for more walltime than remains before the start is not
rejected, it is **deferred to after the window**. Measured against a window
21h49m out, `-t 21:00:00` started immediately and `-t 22:00:00` was pushed
to the reservation's end time — one extra hour of request bought two days of
waiting. The section covers reading `scontrol show reservation`, computing
the gap, confirming with `srun --test-only`, and recognising
`ReqNodeNotAvail, Reserved for maintenance` in `squeue` as "waiting for the
cluster to come back", not something to resubmit. `IGNORE_JOBS` means jobs
already running are not killed when the reservation is created, but nothing
survives the window itself — sinteractive sessions included, so a session
should be launched to end before the start rather than reach past it.

`slurm-discovery` gains the matching `squeue` reason, and a note that
reservations are weather rather than structure: they recur monthly but each
one has a date, so they are read live and never written to the cached map.

- Three more skills covering the things an agent hits in the first ten minutes
of real work on the cluster:

Expand Down
69 changes: 69 additions & 0 deletions skills/bodhi-compute/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,75 @@ them, so the right `-p` can still be rejected under the wrong `-A`. The
`slurm-discovery` skill covers mapping that out, and reading the reason when a
job is refused or stuck `PENDING`.

### Check for a maintenance window before asking for walltime

Bodhi takes a maintenance reservation roughly once a month, and it covers
**every node on the cluster**. Check before requesting anything long:

```bash
scontrol show reservation
# ReservationName=monthly-maint StartTime=2026-08-27T06:00:00
# EndTime=2026-08-28T06:00:00 Duration=1-00:00:00
# Nodes=compgpu[01-03],compute[00-21] NodeCnt=25
# Flags=MAINT,IGNORE_JOBS,SPEC_NODES,ALL_NODES
# Users=root State=INACTIVE
```

`State=INACTIVE` means it has not started yet; `ACTIVE` means it is on and the
nodes are gone. `Users=root` means it is not a reservation you can submit
into. `ALL_NODES` is the part that matters — there is nowhere else to go.

**A job asking for more walltime than remains before the window does not
fail. It gets deferred to after the window.** That is the whole hazard: the
job queues and looks normal, and the cost is invisible unless you check. With
the window 21h49m away:

```console
$ srun --test-only -p rna -c 4 --mem 8G -t 21:00:00 -- true
srun: Job 245072 to start at 2026-08-26T08:10:20 ... # starts now

$ srun --test-only -p rna -c 4 --mem 8G -t 22:00:00 -- true
srun: Job 245073 to start at 2026-08-28T06:00:00 ... # +46 hours
```

One extra hour of requested walltime cost two days of waiting. So **size `-t`
to fit in the gap**, and if the work genuinely cannot fit, say so and let the
user decide between splitting it and waiting.

How long is the gap:

```bash
scontrol show reservation 2>/dev/null |
sed -n 's/.*ReservationName=\([^ ]*\) StartTime=\([^ ]*\) EndTime=\([^ ]*\).*/\1 \2 \3/p' |
while read -r name start end; do
s=$(date -d "$start" +%s) now=$(date +%s)
((s > now)) && printf '%s starts in %dh%02dm\n' \
"$name" $(((s - now) / 3600)) $((((s - now) % 3600) / 60))
done
# monthly-maint starts in 21h49m
```

`--test-only` is the cheap confirmation either way — it reports the start time
the scheduler would actually give the job without queueing anything.

After the fact, a job caught this way says so in `squeue`:

```console
$ squeue -t PENDING -o "%.10i %.12u %.12L %r"
243393 krausmeg 3-00:00:00 ReqNodeNotAvail, Reserved for maintenance
```

Both of those asked for three days with the window a day out. A job in that
state is not stuck and does not need resubmitting — it is waiting for the
cluster to come back. Shortening `-t` is what makes it run sooner.

`IGNORE_JOBS` means the reservation was allowed to be created over jobs that
were already running, so those are not killed up front — but they do not
survive the window either. **Nothing running is safe across it**, sinteractive
sessions included: a session whose walltime crosses the start time will be cut
short, so launch one that ends before the window rather than one that reaches
past it.

## sinteractive sessions

These are the user's persistent interactive shells. You mostly *observe* them;
Expand Down
7 changes: 7 additions & 0 deletions skills/slurm-discovery/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,12 @@ below keeps: what belongs in the file is the structure — which partitions
exist, how big they are, what you may ask for, and the limits on it. Anything
about right now gets run live, every time.

Maintenance reservations are the tempting exception, and they are weather too:
they recur monthly but each one has a date, and a stale window in a file is
exactly the sort of thing that gets trusted. `scontrol show reservation` is
one call — run it, do not cache it. See `bodhi-compute` for what to do with
the answer.

## The survey

**What partitions exist, and how big are they?**
Expand Down Expand Up @@ -159,6 +165,7 @@ The reason names the wall you hit:
| `QOSMaxCpuPerUserLimit`, `AssocMaxJobsLimit` | Your own running jobs are holding the budget. |
| `PartitionTimeLimit`, `PartitionConfig` | The request cannot fit the partition at all. |
| `ReqNodeNotAvail` | Named nodes are down or drained — check `sinfo -R`. |
| `ReqNodeNotAvail, Reserved for maintenance` | `-t` reaches past a maintenance window, so the job is deferred until after it. Shorten it — see `bodhi-compute`. |

For a job already rejected at submit, re-run with `--test-only` to get the
verdict without queueing anything:
Expand Down
Loading