A Mechanistic Study of Transformers Training Dynamics
arXiv:2410.24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models. However, the scale of those models limits our understa