Replica Symmetry Breaking and Algorithmic Thresholds in Empirical Risk Minimization under Multi-Index Model

arXiv:2606.28573v1 Announce Type: new Abstract: Modern machine learning models are trained by optimizing high-dimensional non-convex empirical risk functions. Such cost functions can have a multitude of local optima and yet, gradient-based optimization appears to converge to near-global optima. Within a simple supervised learning setting, we develop a precise picture of which parts of the empirical risk landscape are accessible by polynomial-time algorithms. We are given i.i.d. pairs {(oldsymbol{x}_i,y_i):; 1 le ile n} with oldsymbol{x}_iin R^d standard Gaussian feature vectors, and y_iinR response variables that depend on oldsymbol{x}_i through their projections on an unknown k-dimensional subspace. We use empirical risk minimization to learn a model that depends on an m-dimensional projection of the data (e.g., an m-neurons neural network). We propose an incremental approximate message passing (IAMP) algorithm and precisely characterize the training error it achieves, as well as the relation between test and training error, in the high dimensional asymptotics n,doinfty, with n/doalpha in (0, +infty). Based on earlier work in related models, we expect that the performance achieved by our algorithm is optimal among polynomial-time algorithms.

Source: arXiv cs.LG | 2026-06-30