Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam

arXiv:2607.03998v1 Announce Type: new Abstract: The local sharpness of the loss, the top Hessian eigenvalue lambda_1, determines the largest stable gradient step, but measuring it normally requires Lanczos or Hessian-vector iterations. We observe that a single Armijo backtracking line search already carries this information at the cost of a few forward passes: the accepted step alpha brackets the directional curvature q = g^op H g/|g|^2 within the multiplicative band set by the backtracking factor. Across CIFAR-10, Fashion-MNIST and Imagenette, logalpha tracks loglambda_1 at Pearson -0.91 to -0.95, giving a low-cost online Edge-of-Stability reading. Used once at initialisation, this measurement yields a learning-rate cap (a safeguard, not a faster optimiser) that makes Adam robust to a too-large initial learning rate across more than three orders of magnitude (10^{-3} to 3.0), at about one percent overhead, and it is a no-op when the chosen rate is already safe. One probe is enough: periodic in-training probing adds no robust benefit. The raw-gradient probe exposes the mechanism but needs a safety factor calibrated to the architecture by a one-minute divergence sweep. Probing along Adam's own update direction removes this calibration: a single fixed safety factor kappa = 2 avoids divergence on all nine architectures we test and across the full learning-rate grids of all four benchmarks, and the recipe transfers to AdamW unchanged.

Source: arXiv cs.LG | 2026-07-07