Flat Channels to Infinity in Neural Loss Landscapes

arXiv:2506.14951v4 Announce Type: replace-cross Abstract: The loss landscapes of neural networks contain minima and saddle points that may be connected in flat regions or appear in isolation. We identify and characterize a special structure in the loss landscape: channels along which the loss decreases extremely slowly, while the output weights of at least two neurons, a_i and a_j, diverge to pminfinity, and their input weight vectors, mathbf{w_i} and mathbf{w_j}, become equal to each other. At convergence, the two neurons implement a gated linear unit: a_isigma(mathbf{w_i} dot mathbf{x}) + a_jsigma(mathbf{w_j} dot mathbf{x}) rightarrow sigma(mathbf{w} dot mathbf{x}) + (mathbf{v} dot mathbf{x}) sigma'(mathbf{w} dot mathbf{x}). Geometrically, these channels to infinity are asymptotically parallel to symmetry-induced lines of critical points. Gradient flow solvers, and related optimization methods like SGD or ADAM, reach the channels with high probability in diverse regression settings, but without careful inspection they look like flat local minima with finite parameter values. Our characterization provides a comprehensive picture of these quasi-flat regions in terms of gradient dynamics, geometry, and functional interpretation. The emergence of gated linear units at the end of the channels highlights a surprising aspect of the computational capabilities of fully connected layers.

Source: arXiv cs.AI | 2026-05-11