Activation-Adjacent Layer Normalization: Effects on Approximation Power

arXiv:2610.21527v1 Announce Type: new Abstract: Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore change the function a network represents. We study when shallow activation-normalized (AN) and preactivation-normalized (PN) networks can approximate a prescribed width-N one-hidden-layer MLP. Viewing RMSNorm as input-dependent scaling, we use width augmentation to locally linearize normalization on compact feature sets. For AN networks, an unbounded activation range allows width N+1, while every continuous nonconstant activation admits width N+r with error O_f(r^{-1}). For PN networks, affine parameters or, without them, degree-one positive homogeneity recover width N+1; general continuous activations admit width N+r with error tending to zero. The corresponding LN constructions use centered coordinate pairs, yielding widths N+2 and N+2r, respectively. We prove that N+1 for RMSNorm and N+2 for LN are also the worst-case bound for approximations, and show that applicable universal-approximation theorems for ordinary MLPs transfer to these normalized architectures. Numerical experiments recover the predicted approximation rates for both normalizations across all five settings and illustrate the shifted-ReLU finite-width obstruction.

Source: arXiv cs.LG | 2026-10-16