feat: add server status lights for raven, gannet, and klone

.github/workflows/server-status.yml

@@ -0,0 +1,39 @@
+# Outside-the-network half of the server status lights on the Computing
+# Hardware page.
+#
+# This runner can reach gannet (public HTTPS) and klone (public SSH), but NOT
+# raven, which is not in public DNS. Raven is covered by the cron job inside
+# the UW network -- see scripts/README.md. Running both means a dead internal
+# prober does not take all three lights out at once.
+#
+# GitHub's scheduled workflows are best-effort: they are delayed under load and
+# are disabled automatically after 60 days without repo activity. That is why
+# the in-network cron is the primary prober and this is the backstop.
+name: server-status
+
+on:
+  schedule:
+    - cron: "*/15 * * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+concurrency:
+  group: server-status
+  cancel-in-progress: false
+
+jobs:
+  probe:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Probe public hosts and publish status
+        run: |
+          ./scripts/publish_status.sh \
+            --profile external \
+            --repo "https://x-access-token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" \
+            --workdir "${RUNNER_TEMP}/server-status"
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

docs/Computing-Hardware.md

@@ -6,6 +6,36 @@ Below is a list of computing resources we have available, as well as some links
 
 ---
 
+## Server Status
+
+<table data-status-base="https://raw.githubusercontent.com/RobertsLab/resources/server-status/status">
+  <thead>
+    <tr><th>Server</th><th>Status</th><th>Last checked</th></tr>
+  </thead>
+  <tbody>
+    <tr data-host="raven">
+      <td><code>raven.fish.washington.edu</code></td>
+      <td class="ss-state"><span class="ss-dot ss-unknown"></span><span class="ss-label">checking…</span></td>
+      <td class="ss-time">—</td>
+    </tr>
+    <tr data-host="gannet">
+      <td><code>gannet.fish.washington.edu</code></td>
+      <td class="ss-state"><span class="ss-dot ss-unknown"></span><span class="ss-label">checking…</span></td>
+      <td class="ss-time">—</td>
+    </tr>
+    <tr data-host="klone">
+      <td><code>klone.hyak.uw.edu</code></td>
+      <td class="ss-state"><span class="ss-dot ss-unknown"></span><span class="ss-label">checking…</span></td>
+      <td class="ss-time">—</td>
+    </tr>
+  </tbody>
+</table>
+
+Checks run every 10–15 minutes. Hover a status for details. A reading older than 30 minutes is shown as `unknown` rather than as a stale green light.
+
+!!! note
+    A green light only means the machine answered on the network. It does not mean Slurm is healthy, that there is free disk space, or that your jobs are running. For Hyak-wide outages and maintenance, check [UW-IT Hyak status](https://status.uw.edu/).
+
 ## Accounts
 
 You will need accounts with the following services in order to minimally function in the Roberts Lab:

docs/javascripts/server-status.js

@@ -0,0 +1,141 @@
+/*
+ * Server status lights for the Computing Hardware page.
+ *
+ * Reads JSON written by the probers in scripts/ (see scripts/README.md) and
+ * fills in the status table. Two sources are merged:
+ *
+ *   internal.json - cron on a machine inside the UW network. Only source that
+ *                   can see raven, which is not in public DNS.
+ *   external.json - scheduled GitHub Action. Sees gannet and klone from the
+ *                   public internet. Acts as a backstop if the internal
+ *                   prober host is itself down.
+ *
+ * For each host the most recent fresh reading wins. Anything older than
+ * STALE_MS is reported as unknown rather than shown as a stale green light.
+ */
+(function () {
+  "use strict";
+
+  var STALE_MS = 30 * 60 * 1000; // cron runs every ~10 min; 30 min = missed 3
+  var SOURCES = ["internal.json", "external.json"];
+
+  function fetchSource(base, name) {
+    // Cache-bust: raw.githubusercontent.com serves with max-age=300.
+    var url = base + "/" + name + "?t=" + Date.now();
+    return fetch(url, { cache: "no-store" })
+      .then(function (res) {
+        return res.ok ? res.json() : null;
+      })
+      .catch(function () {
+        return null; // source may not exist yet, or network is down
+      });
+  }
+
+  // Flatten both documents into { host: bestReading }.
+  function mergeReadings(docs) {
+    var best = {};
+    docs.forEach(function (doc) {
+      if (!doc || !doc.hosts) return;
+      var checkedAt = Date.parse(doc.checked);
+      if (isNaN(checkedAt)) return;
+      Object.keys(doc.hosts).forEach(function (host) {
+        var current = best[host];
+        if (current && current.checkedAt >= checkedAt) return;
+        best[host] = {
+          up: doc.hosts[host].up === true,
+          detail: doc.hosts[host].detail || "",
+          latencyMs: doc.hosts[host].latency_ms,
+          source: doc.source || "unknown",
+          checkedAt: checkedAt
+        };
+      });
+    });
+    return best;
+  }
+
+  function relativeTime(then, now) {
+    var mins = Math.round((now - then) / 60000);
+    if (mins < 1) return "just now";
+    if (mins === 1) return "1 min ago";
+    if (mins < 60) return mins + " min ago";
+    var hrs = Math.round(mins / 60);
+    if (hrs === 1) return "1 hr ago";
+    if (hrs < 24) return hrs + " hr ago";
+    var days = Math.round(hrs / 24);
+    return days === 1 ? "1 day ago" : days + " days ago";
+  }
+
+  function paintRow(row, reading, now) {
+    var stateCell = row.querySelector(".ss-state");
+    var timeCell = row.querySelector(".ss-time");
+    if (!stateCell || !timeCell) return;
+
+    var cls, label, title;
+
+    if (!reading) {
+      cls = "ss-unknown";
+      label = "unknown";
+      title = "No status data available for this host yet.";
+      timeCell.textContent = "—";
+    } else if (now - reading.checkedAt > STALE_MS) {
+      cls = "ss-unknown";
+      label = "unknown";
+      title =
+        "Last reading is stale (" +
+        relativeTime(reading.checkedAt, now) +
+        ") — the prober itself may be down.";
+      timeCell.textContent = relativeTime(reading.checkedAt, now);
+    } else {
+      cls = reading.up ? "ss-up" : "ss-down";
+      label = reading.up ? "up" : "not responding";
+      title =
+        reading.detail +
+        (reading.latencyMs != null ? " (" + reading.latencyMs + " ms)" : "") +
+        " — checked by " +
+        reading.source +
+        " prober";
+      timeCell.textContent = relativeTime(reading.checkedAt, now);
+    }
+
+    stateCell.innerHTML = "";
+    var dot = document.createElement("span");
+    dot.className = "ss-dot " + cls;
+    var text = document.createElement("span");
+    text.className = "ss-label";
+    text.textContent = label; // never color alone: the word carries the meaning
+    stateCell.appendChild(dot);
+    stateCell.appendChild(text);
+    stateCell.title = title;
+  }
+
+  function render() {
+    var table = document.querySelector("[data-status-base]");
+    if (!table) return;
+    var base = table.getAttribute("data-status-base");
+
+    Promise.all(
+      SOURCES.map(function (name) {
+        return fetchSource(base, name);
+      })
+    ).then(function (docs) {
+      var readings = mergeReadings(docs);
+      var now = Date.now();
+      var rows = table.querySelectorAll("tr[data-host]");
+      Array.prototype.forEach.call(rows, function (row) {
+        paintRow(row, readings[row.getAttribute("data-host")], now);
+      });
+    });
+  }
+
+  if (document.readyState === "loading") {
+    document.addEventListener("DOMContentLoaded", render);
+  } else {
+    render();
+  }
+
+  // mkdocs-material swaps page content without a reload when instant
+  // navigation is enabled, so re-render on those transitions too.
+  if (window.document$ && typeof window.document$.subscribe === "function") {
+    window.document$.subscribe(render);
+  }
+})();

docs/stylesheets/server-status.css

@@ -0,0 +1,49 @@
+/* Status lights for the Computing Hardware page. */
+
+.ss-dot {
+  display: inline-block;
+  width: 0.7em;
+  height: 0.7em;
+  margin-right: 0.45em;
+  border-radius: 50%;
+  vertical-align: baseline;
+  background: var(--ss-color, #9aa0a6);
+  box-shadow: 0 0 0 1px rgba(0, 0, 0, 0.12) inset;
+}
+
+.ss-up {
+  --ss-color: #2e9e4f;
+}
+
+.ss-down {
+  --ss-color: #d64545;
+}
+
+.ss-unknown {
+  --ss-color: #9aa0a6;
+}
+
+[data-md-color-scheme="slate"] .ss-up {
+  --ss-color: #4ad07d;
+}
+
+[data-md-color-scheme="slate"] .ss-down {
+  --ss-color: #ff6b6b;
+}
+
+[data-md-color-scheme="slate"] .ss-unknown {
+  --ss-color: #8a8f98;
+}
+
+.ss-label {
+  font-variant-numeric: tabular-nums;
+}
+
+.ss-state {
+  white-space: nowrap;
+}
+
+.ss-time {
+  white-space: nowrap;
+  opacity: 0.75;
+}

mkdocs.yml

@@ -27,6 +27,12 @@ theme:
       toggle:
         icon: material/toggle-switch
         name: Switch to light mode
+extra_css:
+  - stylesheets/server-status.css
+
+extra_javascript:
+  - javascripts/server-status.js
+
 markdown_extensions:
   - admonition
   - pymdownx.details

scripts/README.md

@@ -0,0 +1,106 @@
+# Server status lights
+
+These scripts drive the status table at the top of
+[Computing Hardware](../docs/Computing-Hardware.md).
+
+## Why there are two probers
+
+The three servers are not reachable from the same place:
+
+| Host | Public DNS | How it is checked |
+|---|---|---|
+| `raven.fish.washington.edu` | no | TCP 8787 (RStudio Server), falling back to 22 — **UW network only** |
+| `gannet.fish.washington.edu` | yes | HTTPS `GET /` |
+| `klone.hyak.uw.edu` | yes | TCP 22 — there is no web port to probe |
+
+Raven is not in public DNS, so nothing outside the UW network can see it. Klone
+has no HTTP port at all, and browsers refuse to connect to port 22, so no
+purely client-side check can work either.
+
+So there are two probers writing to the same place:
+
+- **`internal`** — cron on a machine inside the UW network. The only source
+  that can see raven. This is the primary.
+- **`external`** — the `server-status` GitHub Action. Sees gannet and klone
+  from the public internet, so a dead internal prober does not take out all
+  three lights at once.
+
+Each writes `status/<profile>.json` to the orphan `server-status` branch. The
+page fetches both from `raw.githubusercontent.com` (which sends
+`access-control-allow-origin: *`, so there is no CORS problem) and takes the
+most recent fresh reading per host.
+
+The branch is rewritten as a single root commit on every run. At one check
+every 10 minutes an append-only branch would add roughly 50,000 commits a year
+to a repo that everyone clones.
+
+## Files
+
+- `check_servers.py` — runs the probes, prints or writes the JSON. Stdlib only.
+- `publish_status.sh` — runs the checker and pushes the result to the
+  `server-status` branch. Used by both the cron job and the Action.
+
+Check without publishing anything:
+
+```bash
+./scripts/check_servers.py --profile internal
+```
+
+Off the UW network, raven will report `DNS lookup failed` — that is expected,
+and is exactly why the internal prober has to run inside.
+
+## Setting up the in-network cron
+
+Run this on a machine inside the UW network that is up continuously. Gannet is
+the natural choice.
+
+**1. Give the machine push access.** Generate a deploy key on that host:
+
+```bash
+ssh-keygen -t ed25519 -f ~/.ssh/robertslab_status -C "roberts-lab status bot" -N ""
+```
+
+Add the public key (`~/.ssh/robertslab_status.pub`) at
+<https://github.com/RobertsLab/resources/settings/keys> as a deploy key **with
+write access checked**. A deploy key is scoped to this one repo, so it cannot
+be used to touch anything else in the org.
+
+Then point git at it in `~/.ssh/config`:
+
+```
+Host github-robertslab
+  HostName github.com
+  User git
+  IdentityFile ~/.ssh/robertslab_status
+  IdentitiesOnly yes
+```
+
+**2. Clone the repo** somewhere on that host, e.g. `~/robertslab-resources`.
+
+**3. Add the cron entry** with `crontab -e`:
+
+```
+*/10 * * * * ~/robertslab-resources/scripts/publish_status.sh --profile internal --repo github-robertslab:RobertsLab/resources.git >> ~/status-cron.log 2>&1
+```
+
+`publish_status.sh` keeps its own scratch clone under
+`~/.cache/robertslab-server-status`, so it will not disturb the checkout it is
+run from.
+
+**4. Confirm** that `status/internal.json` appears on the
+[`server-status` branch](https://github.com/RobertsLab/resources/tree/server-status/status)
+and that the lights on the handbook page go green within a few minutes.
+
+## Notes and limits
+
+- A green light means the port answered. It says nothing about Slurm health,
+  disk space, or whether jobs are running.
+- `raw.githubusercontent.com` caches for about 5 minutes, so the page can lag
+  the actual check by that much on top of the check interval.
+- GitHub's scheduled workflows are best-effort: delayed under load, minimum
+  5-minute interval, and disabled automatically after 60 days without repo
+  activity. That is why the in-network cron is primary and the Action is only a
+  backstop.
+- To add a host: add a probe in `check_servers.py` and a `<tr data-host="...">`
+  row in `docs/Computing-Hardware.md`. The JavaScript matches the two by name
+  and needs no change.