Towards Stability of Parameter-Free Optimization

arXiv:2405.04376v4 Announce Type: replace Abstract: Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge. To address this challenge, we propose a novel parameter-free optimizer, extsc{AdamG} (Adam with the Golden step size), designed to automatically adapt to diverse optimization problems without task-specific learning-rate tuning. The core technique underlying extsc{AdamG} is our golden step size derived for the AdaGrad-Norm algorithm, which is expected to help AdaGrad-Norm preserve tuning-free convergence and approximate the optimal step size in expectation across diverse optimization scenarios. To better evaluate tuning-free performance, we propose a novel evaluation criterion, extit{reliability}, to comprehensively assess the efficacy of parameter-free optimizers in addition to classical performance criteria. Empirical results demonstrate that extsc{AdamG} outperforms other parameter-free baselines and consistently performs on par with Adam using a manually tuned learning rate across various optimization tasks.

Related

• Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
• POP: Prior-Fitted First-Order Optimization Policies
• Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

Source: arXiv cs.LG | 2026-07-31