Renyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization

arXiv:2510.07758v3 Announce Type: replace Abstract: Sharpness (of the loss minima) is widely believed to be a good indicator of generalization of neural networks. Unfortunately, the correlation between existing sharpness measures and generalization is not as strong as expected, and sometimes even contradiction occurs. To address this problem, a key observation in this paper is: what really matters for generalization is the average spread (or unevenness) of the spectrum of loss Hessian mathbf{H}. For this reason, conventional sharpness measures, such as trace sharpness operatorname{tr}(mathbf{H}), which cares about the average value of the spectrum, or max-eigenvalue sharpness lambda_{max}(mathbf{H}), which concerns the maximum spread of the spectrum, are not sufficient to well predict generalization. To characterize the average spread of the Hessian spectrum, we leverage the notion of Renyi entropy in information theory, which captures the unevenness of a probability vector and can thus be extended to a general non-negative vector, such as the Hessian spectrum at loss minima. Specifically, we propose Renyi sharpness, defined as the negative of the Renyi entropy of loss Hessian mathbf{H}. Extensive experiments demonstrate that Renyi sharpness exhibits strong and consistent correlation with generalization in various scenarios. Moreover, two generalization bounds with respect to Renyi sharpness are established by exploiting its desirable reparametrization invariance property. Finally, as an initial attempt to exploit Renyi sharpness for regularization, Renyi Sharpness Aware Minimization (RSAM) is proposed, where a variant of Renyi sharpness is used as the regularizer. RSAM is competitive with state-of-the-art SAM algorithms and far better than conventional SAM based on max-eigenvalue sharpness.

Source: arXiv cs.LG | 2026-08-21