Research
Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection
arXiv:2608.18810v1 Announce Type: new Abstract: An optimizer is usually chosen before training a deep neural network and then kept fixed. Treating optimizer choice as a hyperparameter could boost perf
arXiv:2608.18810v1 Announce Type: new Abstract: An optimizer is usually chosen before training a deep neural network and then kept fixed. Treating optimizer choice as a hyperparameter could boost performance, but it requires several complete training runs and discards all but the winner. Repeated Optimizer Resampling (ROR) instead searches during one evolving run. Every b epochs, each candidate optimizer scouts from the current model weights for s epochs. The best scout continues for the remaining b-s epochs, and that completed segment becomes the new incumbent if it improves the validation objective. This design allows the preferred optimizer to change as training progresses. We compare two variants of ROR on MNIST, Fashion-MNIST, and two motor insurance claim-count models. Nine fixed optimizers and both ROR variants are evaluated with the same ten seeds. One-epoch ROR uses 24% to 35% of the aggregate training needed to identify the best fixed optimizer exhaustively and remains close to that optimizer on all four tasks. These results support short scouting as a practical way to search over optimizers without completing every candidate run.
Related
- On the Principles Behind Neural Network Optimizers
- AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization
- Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimization
- ZENITH: Automated Gradient Norm Informed Stochastic Optimization
Source: arXiv cs.LG | 2026-08-20