Skip to content

feat: add server status lights for raven, gannet, and klone - #2484

Merged
sr320 merged 1 commit into
masterfrom
server-status-lights
Aug 8, 2026
Merged

feat: add server status lights for raven, gannet, and klone#2484
sr320 merged 1 commit into
masterfrom
server-status-lights

Conversation

@sr320

@sr320 sr320 commented Aug 8, 2026

Copy link
Copy Markdown
Member

Adds an at-a-glance status table to the top of the Computing Hardware page for raven, gannet, and klone.

Why it is built this way

The three hosts are not reachable from the same place, and that drives the whole design:

Host Public DNS How it is checked
raven.fish.washington.edu no TCP 8787 (RStudio Server), falling back to 22 — UW network only
gannet.fish.washington.edu yes HTTPS GET /
klone.hyak.uw.edu yes TCP 22 — there is no web port to probe

Raven is not in public DNS, so nothing outside the UW network can see it. Klone has no HTTP port at all, and browsers refuse to connect to port 22 as an unsafe port. So a purely client-side check cannot cover all three.

Instead, two probers write status/<profile>.json to an orphan server-status branch:

  • internal — cron inside the UW network. The only source that can see raven. Primary.
  • external — scheduled GitHub Action. Covers the two public hosts, so a dead internal prober does not take out all three lights at once.

The page fetches both from raw.githubusercontent.com (which sends access-control-allow-origin: *, so no CORS problem) and takes the most recent fresh reading per host.

Two deliberate choices worth calling out:

  • Stale readings render as unknown, not green. Anything older than 30 minutes is greyed out, so a dead prober is visible rather than silently reporting everything up.
  • The status branch is rewritten as a single root commit each run. At one check every 10 minutes, an append-only branch would add roughly 50,000 commits a year to a repo everyone clones. Concurrent probers use --force-with-lease, so a race is rejected and retried instead of clobbering the other prober's file.

Verification

Built the site with mkdocs-material and drove the rendered page in a browser:

  • Live probes correct: gannet HTTP 200, klone TCP 22 open in 9 ms, raven DNS lookup failed from off-network — exactly why the internal prober is required.
  • Merge logic: with internal reporting klone down and external reporting it up, the fresher internal reading won and the light went red.
  • Fed it 2-hour-old JSON with up: true on every host; all three correctly rendered grey unknown.
  • Missing data (branch not yet created, both fetches 404) degrades to unknown with no console errors.
  • Dark mode resolves the slate dot colors. The status word accompanies every dot, so meaning never rests on color alone.
  • publish_status.sh survives repeat runs and holds the branch at exactly one commit across cycles; raced two workdirs and confirmed the loser was rejected with stale info while the winner's file survived.

Follow-up needed before raven reports

  1. Merging this — on: schedule only fires from the default branch, so the external prober is dormant until then.
  2. Setting up the in-network cron on gannet (deploy key, clone, one crontab line). Steps in scripts/README.md. Until then raven reads unknown; gannet and klone light up from the Action alone.

Note

A green light means the port answered. It says nothing about Slurm health, disk space, or whether jobs are running — stated on the page itself.

The existing > [!IMPORTANT] block on this page renders as literal text because the repo lacks the github-callouts extension. I used !!! note for the new callout so it renders correctly, rather than widening this PR to fix the pre-existing ones.

🤖 Generated with Claude Code

Adds an at-a-glance status table for raven, gannet, and klone at the top
of the Computing Hardware page.

The three hosts are not reachable from the same place, which shapes the
design:

  raven   not in public DNS; visible only from inside the UW network
  gannet  public HTTPS
  klone   public, but SSH only -- no web port to probe, and browsers
          refuse to connect to port 22

So no purely client-side check can cover all three. Instead two probers
write status/<profile>.json to an orphan `server-status` branch:

  internal  cron inside the UW network; the only source that sees raven
  external  scheduled GitHub Action; covers the two public hosts so a
            dead internal prober does not take out all three lights

The page fetches both from raw.githubusercontent.com (which sends
access-control-allow-origin: *) and takes the most recent fresh reading
per host. Readings older than 30 minutes render as "unknown" rather than
as a stale green light, so a dead prober is visible instead of silently
reporting everything up.

The status branch is rewritten as a single root commit each run; at one
check every 10 minutes an append-only branch would add ~50k commits a
year to a repo everyone clones. Concurrent probers are handled with
--force-with-lease, so a race is rejected and retried rather than
clobbering the other prober's file.

The in-network cron still needs to be set up on gannet before raven
reports; see scripts/README.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sr320
sr320 merged commit 8a375cb into master Aug 8, 2026
2 checks passed
@sr320
sr320 deleted the server-status-lights branch August 8, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant