Tauon: A new optimizer outperforming Muon on GPT-Mini (lower loss, ~8.5% faster step time) [P]
Hey r/MachineLearning! I’ve been working on a new optimizer called Tauon (turns out there is already 'teon' but well if you have better idea, - i will gladly accept it! Anyway the core idea of optimiz