Add FlashSAC agent for high-throughput off-policy training

[tmjeong1103]

Summary

• Add a MimicKit-native FlashSAC agent, model, and replay buffer.
• Integrate FlashSAC with MimicKit's agent builder, vector-environment lifecycle, evaluation, and checkpoint format.
• Add a memory-reduced DeepMimic G1 example, documentation, tests, and upstream MIT attribution.

Implementation notes

• The actor and critic consume raw observations, matching the reference FlashSAC implementation.
• Policy actions operate in [-1, 1] and use MimicKit's action normalizer to map to the environment bounds.
• FAIL and SUCC are terminal states; TIME is treated as a truncation so the critic can bootstrap.
• The update budget is accumulated per environment transition to keep the update-to-data ratio stable across vector-environment counts.
• Checkpoints include the model, optimizer, schedulers, reward normalizer, and training counters. Replay-buffer sidecars are opt-in because they can be large.
• The eager CUDA path is the validated compatibility path. torch.compile remains opt-in because its behavior is PyTorch-version dependent.

The implementation follows the public FlashSAC PyTorch repository at commit 87edc9061150ae9e962dd84e6544e27a1554b3ab while adapting environment interaction and persistence to MimicKit conventions.

Validation

python -m unittest discover -s tests -p "test_flashsac*.py" -v

• 26/26 FlashSAC unit tests pass.
• git diff --check passes.
• Deterministic policies were trained and evaluated successfully on three G1 DeepMimic motions.

One-hour wall-clock comparison

The following is a single-seed descriptive comparison using 4,096 environments on one RTX 3090 Ti. Both agents received the same one-hour wall-clock budget. Reported values use deterministic 256-environment evaluation and causal wall-clock alignment.

Motion	30 min return (FlashSAC / PPO)	Common-horizon final return (FlashSAC / PPO)	Time-average return (FlashSAC / PPO)
G1 kick combo	196.83 / 36.55	277.69 / 207.02	183.22 / 80.52
G1 speed vault	25.82 / 14.55	26.75 / 24.13	23.27 / 15.06
G1 cartwheel	165.61 / 36.74	278.58 / 221.51	172.57 / 73.91

This benchmark is intended as integration evidence rather than a statistical-significance or exact paper-reproduction claim.

G1 kick combo

G1 speed vault

G1 cartwheel

Example

python mimickit/run.py \
    --arg_file args/deepmimic_g1_flashsac_args.txt \
    --mode train \
    --visualize false

The included example reduces replay capacity and uses n_step: 1 to lower memory requirements. The documentation distinguishes these choices from the larger settings used by the reference IsaacLab launcher.

References

• Paper: FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control
• Reference implementation: Holiday-Robot/FlashSAC
• Reference commit: 87edc9061150ae9e962dd84e6544e27a1554b3ab

Attribution

The original FlashSAC MIT license is included at LICENSES/FlashSAC-MIT.txt, and the documentation includes the paper citation and source commit used for the adaptation.