Add health checks for nvidia and AMD (#315)

* nvidia runner health check

* add amd

* Update amd-health.yml

* Update nvidia-arc-health.yml

Author: Mark Saroufim

.github/workflows/amd-health.yml

@@ -0,0 +1,27 @@
+name: amd
+
+on:
+  schedule:
+    # Run nightly at 2 AM UTC
+    - cron: '0 2 * * *'
+  workflow_dispatch:
+  push:
+    branches: [main]
+
+jobs:
+  health-check:
+    runs-on: [amdgpu-mi300-x86-64]
+    timeout-minutes: 5
+    
+    steps:
+    - name: Setup Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.10'
+    
+    - name: Install PyTorch
+      run: |
+        pip install torch
+    
+    - name: GPU Health Check
+      run: python -c "import torch; torch.randn(5, device='cuda')"

.github/workflows/nvidia-arc-health.yml

@@ -0,0 +1,32 @@
+name: nvidia-arc
+
+on:
+  schedule:
+    # Run nightly at 2 AM UTC
+    - cron: '0 2 * * *'
+  workflow_dispatch:
+  push:
+    branches: [main]
+
+jobs:
+  health-check:
+    runs-on: [gpumode-nvidia-arc]
+    timeout-minutes: 5
+    container:
+      image: nvidia/cuda:12.4.0-devel-ubuntu22.04
+    
+    steps:
+    - name: Setup Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.10'
+    
+    - name: Install PyTorch
+      run: |
+        pip install torch
+    
+    - name: GPU Health Check
+      run: python -c "import torch; torch.randn(5, device='cuda')"
+    
+    env:
+      CUDA_VISIBLE_DEVICES: 0

README.md

@@ -1,5 +1,8 @@
 # discord-cluster-manager
 
+[![nvidia-arc](https://github.com/pytorch-labs/discord-cluster-manager/actions/workflows/nvidia-arc-health.yml/badge.svg)](https://github.com/pytorch-labs/discord-cluster-manager/actions/workflows/nvidia-arc-health.yml)
+[![amd](https://github.com/pytorch-labs/discord-cluster-manager/actions/workflows/amd-health.yml/badge.svg)](https://github.com/pytorch-labs/discord-cluster-manager/actions/workflows/amd-health.yml)
+
 This is the code for the Discord bot we'll be using to queue jobs to a cluster of GPUs that our generous sponsors have provided. Our goal is to be able to queue kernels that can run end to end in seconds that way things feel interactive and social.
 
 The key idea is that we're using Github Actions as a job scheduling engine and primarily making the Discord bot interact with the cluster via issuing Github Actions and and monitoring their status and while we're focused on having a nice user experience on discord.gg/gpumode, [we're happy to accept PRs](#local-development) that make it easier for other Discord communities to hook GPUs.