plaiground is a beautiful, full-stack visual interface for AI agents to interact with Gymnasium environments and retro emulators in real-time.
- Full-Stack Architecture: A fast Python FastAPI backend paired with a stunning Vite/React frontend.
- Real-Time Simulation: WebSockets provide high-performance streaming of environment frames and agent actions directly to your browser.
- Pluggable Architecture: Easily swap out environments (Gymnasium, stable-retro) or agent logic (Random, Heuristic, LLMs).
- Open a terminal and navigate to the backend directory:
cd backend_python
Warning
Windows Users: The stable-retro library requires compilation of C++ emulation cores and is not natively supported on Windows. You must run the backend within WSL (Windows Subsystem for Linux) or a native Linux environment to successfully install the dependencies and run the emulator.
- Install dependencies:
pip install -r requirements.txt
- Start the backend server:
python main.py
- Open a new terminal and navigate to the frontend directory:
cd frontend - Install dependencies:
npm install
- Start the Vite dev server:
npm run dev
- Open your browser to the URL provided (typically
http://localhost:5173).
Phase 1 & Phase 2 are complete!
The current version implements a robust Python backend integrated with Gymnasium and stable-retro, streaming base64 frames to a React UI in real-time.
- Real-Time Simulation: WebSockets provide high-performance streaming of environment frames and agent actions directly to your browser.
- Gymnasium & Stable-Retro Emulation: Easily swap between standard RL environments (CartPole, etc.) and full libretro/GBA emulation (e.g., Pokémon FireRed) directly from the UI.
- Custom Modded ROM Support: Built-in helper script (
setup_custom_rom.py) to easily package and run modified ROM files bypassing standard checksum validations. - Vision LLM Agents: Connects out-of-the-box to Google Gemini, OpenAI, Anthropic, or local (Ollama/GPT4All) Large Language Models as agents that visually interpret frames and issue commands.
- Decoupled Simulation Speeds: The orchestrator allows independent tuning of Emulator Speed (FPS) and Agent Action Frequency, utilizing a "Tap & Wait" strategy to allow complex game animations to play out naturally without spamming LLM requests.
- Time-Travel Debugging: Implement full state recording, allowing users to pause, rewind, and replay simulations step-by-step from the UI.
- Metrics Dashboard: Add rich charting libraries to graph agent performance (cumulative rewards, survival time) over thousands of episodes.
- Visual Saliency Maps: Display what the agent is "looking at" or focusing on during its decision-making process.