Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates

Preliminary code release for our paper "Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates", by Athanasios Glentis, Dawei Li, Chung-Yiu Yau and Mingyi Hong.