Why Adaptive Optimizers Underestimate Rare Tokens
DGX agentarXiv:2609.37535v1 Announce Type: new Abstract: In the softmax output layer, a rare token receives a small positive logit gradient on most steps and a much larger negative gradient on the few steps wh