feat: add server status lights for raven, gannet, and klone - #2484
Merged
Conversation
Adds an at-a-glance status table for raven, gannet, and klone at the top
of the Computing Hardware page.
The three hosts are not reachable from the same place, which shapes the
design:
raven not in public DNS; visible only from inside the UW network
gannet public HTTPS
klone public, but SSH only -- no web port to probe, and browsers
refuse to connect to port 22
So no purely client-side check can cover all three. Instead two probers
write status/<profile>.json to an orphan `server-status` branch:
internal cron inside the UW network; the only source that sees raven
external scheduled GitHub Action; covers the two public hosts so a
dead internal prober does not take out all three lights
The page fetches both from raw.githubusercontent.com (which sends
access-control-allow-origin: *) and takes the most recent fresh reading
per host. Readings older than 30 minutes render as "unknown" rather than
as a stale green light, so a dead prober is visible instead of silently
reporting everything up.
The status branch is rewritten as a single root commit each run; at one
check every 10 minutes an append-only branch would add ~50k commits a
year to a repo everyone clones. Concurrent probers are handled with
--force-with-lease, so a race is rejected and retried rather than
clobbering the other prober's file.
The in-network cron still needs to be set up on gannet before raven
reports; see scripts/README.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an at-a-glance status table to the top of the Computing Hardware page for
raven,gannet, andklone.Why it is built this way
The three hosts are not reachable from the same place, and that drives the whole design:
raven.fish.washington.edugannet.fish.washington.eduGET /klone.hyak.uw.eduRaven is not in public DNS, so nothing outside the UW network can see it. Klone has no HTTP port at all, and browsers refuse to connect to port 22 as an unsafe port. So a purely client-side check cannot cover all three.
Instead, two probers write
status/<profile>.jsonto an orphanserver-statusbranch:The page fetches both from
raw.githubusercontent.com(which sendsaccess-control-allow-origin: *, so no CORS problem) and takes the most recent fresh reading per host.Two deliberate choices worth calling out:
unknown, not green. Anything older than 30 minutes is greyed out, so a dead prober is visible rather than silently reporting everything up.--force-with-lease, so a race is rejected and retried instead of clobbering the other prober's file.Verification
Built the site with mkdocs-material and drove the rendered page in a browser:
DNS lookup failedfrom off-network — exactly why the internal prober is required.up: trueon every host; all three correctly rendered greyunknown.unknownwith no console errors.publish_status.shsurvives repeat runs and holds the branch at exactly one commit across cycles; raced two workdirs and confirmed the loser was rejected withstale infowhile the winner's file survived.Follow-up needed before raven reports
on: scheduleonly fires from the default branch, so the external prober is dormant until then.scripts/README.md. Until then raven readsunknown; gannet and klone light up from the Action alone.Note
A green light means the port answered. It says nothing about Slurm health, disk space, or whether jobs are running — stated on the page itself.
The existing
> [!IMPORTANT]block on this page renders as literal text because the repo lacks the github-callouts extension. I used!!! notefor the new callout so it renders correctly, rather than widening this PR to fix the pre-existing ones.🤖 Generated with Claude Code