Tutorials
Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks
arXiv:2610.20069v1 Announce Type: new Abstract: What makes a deep network train, and what shape is its loss surface? We answer both with one construction at two scales: a polynomial activation f (z) =
arXiv:2610.20069v1 Announce Type: new Abstract: What makes a deep network train, and what shape is its loss surface? We answer both with one construction at two scales: a polynomial activation f (z) = z + {lambda}z 3 , clamped and projected onto the unit sphere, plus an unnormalized identity path. As a deep stack it trains to one hundred layers on MNIST with no normalization machinery; as a small model its square loss is a polynomial system we solve exactly. The small scale explains the large one: it prescribes how to design, initialize and train the stack.
Related
- Stable initialization without the CLT
- Benign Loss Landscapes Can Coexist with Worst-Case Hardness
- Shallow Neural Networks Learn Low-Degree Spherical Polynomials with Feature Learning by Learnable Channel Attention
- FlexAct: Why Learn when you can Pick?
Source: arXiv cs.LG | 2026-10-15