Width-Independent Compressibility of Deep Neural Networks

arXiv:2608.21752v1 Announce Type: new Abstract: It has long been known that well-trained neural networks can be compressed very strongly without affecting their performance, an important phenomenon that remains poorly understood. We prove a uniform compressibility theorem for deep multilayer perceptrons with analytic activations. For a deep, wide fixed teacher network, there exists a narrow (same depth) network that approximately represents the same function as the original. The reachable compressed width is strikingly independent of the original width, but is O((log(1/arepsilon))^{d_{in}}), where arepsilon is the error budget and d_{in} is the effective input dimension. Our construction involves a novel derivative-matching technique which is aware of the low-dimensional input, and a layer-wise reweighting that preserves the input-output mapping.

Source: arXiv cs.LG | 2026-08-25