[GUIDE / FIX] Fixing sudden speed drops (3 T/s bug) on Multi-GPU Windows rigs running 100B+ MoE models on CUDA 13

[AndyAnttle]

Hey everyone,
If you are running large MoE models (like DeepSeek V4, GLM-4.5, or Laguna) on Windows 10 with a Multi-GPU setup (e.g., 4x3090) and experiencing a brutal slowdown down to ~3 T/s after the first generation, I found a clean and working solution.

The Problem:
On recent NVIDIA drivers with CUDA 13 support, the Windows WDDM driver and CUDA Memory Manager become "too smart." When a huge model fills your VRAM near 95-98% and goes idle between prompts, CUDA 13 aggressively and silently triggers Shared System Memory Spilling. It moves parts of your MoE layers/routers to system RAM over slow PCIe channels to prevent system crashes. Opening any GUI element (like Kobold settings window) temporarily wakes it up, forcing a defrag, but then it throttles again.
Tweaking "Prefer No Sysmem Fallback" in NVIDIA Control Panel doesn't always work if you have asymmetric cards (like 3090s + a 3060 display adapter).

The Fix (The CUDA Heartbeat Script):
The trick is to trick CUDA into thinking the GPUs are constantly busy so it never triggers the power-saving P-state drop or the pre-emptive RAM offload. We can achieve this by running a lightweight background Python script that sends a micro-calculation to every CUDA device every second. It uses less than 0.1% of performance but keeps the context alive.
Install PyTorch with CUDA support.
Create a script named cuda_heartbeat.py:

python

import torch
import time
import sys

# Initialize context on all available GPUs
devices = [torch.device(f"cuda:{i}") for i in range(torch.cuda.device_count())]
print(f"[*] CUDA Heartbeat initialized on {len(devices)} GPUs. Keeping them alive...")

try:
    while True:
        for device in devices:
            # Tiny matrix multiplication to keep the tensor cores awake
            x = torch.randn(10, 10, device=device)
            y = torch.matmul(x, x)
            torch.cuda.synchronize(device)
        
        time.sleep(1.0) # Sleep for 1 second to avoid CPU overhead
except KeyboardInterrupt:
    print("\n[*] Heartbeat stopped.")

How to use:
Run this script in a separate command prompt window, then launch your KoboldCPP / Llama.cpp with your 100B+ MoE model fully offloaded to VRAM (--gpulayers 44, --n-cpu-moe 0).
My speed went from a dying 3.5 T/s back to a flawless 16+ T/s on subsequent generations (GLM 4.5 and DeepSeek V4 Flash UD-Q2_M on 4x3090 setup) without messing with clock locks or dual GUI windows.
Hope this helps anyone stuck on Windows!

Note: This solution and script were co-developed with Google Gemini during a deep-dive debugging session regarding Windows WDDM and CUDA 13 memory allocation bugs.