CleanMARL

CleanMARL provides single-file, clean, and educational implementations of Deep Multi-Agent Reinforcement Learning (MARL) algorithms in PyTorch, following the same philosophy of CleanRL.

Main Features

• Implementations of key MARL algorithms: VDN, QMIX, COMA, MADDPG, FACMAC, IPPO, and MAPPO.

• A documentation for algorithms, code and training details.

• We support continuous and discrete actions.

• We support parallel environments and recurrent policies.

• Tensorboard and Weights & Biases logging.

We provide more details in our documentation.

Check the old_jax branch for JAX implementations (non-jax envs only).

Quick Start

Prerequisites:

• Python >=3.9

Installation:

git clone https://github.com/AmineAndam04/cleanmarl.git
cd cleanmarl
pip install .

To run experiment you can run for example:

python  cleanmarl/vdn.py --env_type="pz" --env_name="simple_spread_v3" --env_family="mpe" --use_wnb --wnb_project="cleanmarl-test" --wnb_entity="cleanmarl-test" --total_timesteps=1000000

python  cleanmarl/mappo.py --env_type="smaclite" --env_name="3m" 

Supported Algorithms

Algorithm	Variants Implemented
Value Decomposition Networks (VDN)	vdn.py vdn_lstm.py vdn_multienvs.py
QMIX	qmix.py qmix_lstm.py qmix_multienvs.py
Counterfactual Multi-Agent (COMA)	coma.py coma_lstm.py coma_multienvs.py coma_lstm_multienvs.py
Multi-Agent Deep Deterministic Policy Gradient (MADDPG)	maddpg.py maddpg_multienvs.py maddpg_lstm.py maddpg_continuous
Factored Multi-Agent Centralized Policy Gradients (FACMAC)	facmac.py facmac_multienvs.py facmac_continuous
Independent Proximal Policy Optimization (IPPO)	ippo.py ippo_lstm.py ippo_multienvs.py ippo_lstm_multienvs.py ippo_continuous
Multi-Agent Proximal Policy Optimization (MAPPO)	mappo.py mappo_lstm.py mappo_multienvs.py mappo_lstm_multienvs.py mappo_continuous

Supported environments

We use marlbench to interact with MARL environments. marlbench is a tool that provides (1) a common API for MARL envs, (2) vectorized envs, and (2) common wrappers (normalization, clipping ...)

Install using uv pip install marlbench

• Git repo: marlbench

Environment	Action space	Installation
Level-Based Foraging	Discrete	pip install lbforaging
Multi-Robot Warehouse	Discrete	pip install rware
SMAClite	Discrete	Install it from its GitHub repository
PettingZoo	Discrete or continuous	pip install pettingzoo and install the extra dependencies for the family you use
MaMuJoCo	Continuous	pip install gymnasium-robotics
MAgent2	Discrete	pip install magent2
SMAC	Discrete	Follow the instructions in the SMAC repository
SMACv2	Discrete	Follow the instructions in the SMACv2 repository