Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization

arXiv:2604.03146v2 Announce Type: replace-cross Abstract: We study high-dimensional convex empirical risk minimization (ERM) under general non-Gaussian data designs. By heuristically extending the Convex Gaussian Min-Max Theorem (CGMT) to non-Gaussian settings, we derive an asymptotic min-max characterization of key statistics, enabling approximation of the mean mu_{hat{heta}} and covariance C_{hat{heta}} of the ERM estimator hat{heta}. Specifically, under a concentration assumption on the data matrix and standard regularity conditions on the loss and regularizer, we show that for a test covariate x independent of the training data, the projection hat{heta}^op x approximately follows the convolution of the generally non-Gaussian distribution of mu_{hat{heta}}^op x with an independent centered Gaussian variable of variance tr(C_{hat{heta}} E[xx^op]). This result clarifies the scope and limits of Gaussian universality for ERMs. Additionally, we prove that any C^2 regularizer is asymptotically equivalent to a quadratic form determined solely by its Hessian at zero and gradient at mu_{hat{heta}}. Numerical simulations across diverse losses and models are provided to validate our theoretical predictions and qualitative insights.

Source: arXiv cs.LG | 2026-06-08