Sharp Approximation Rates for Neural Networks with Affine Latent Parameterizations

arXiv:2608.31157v1 Announce Type: new Abstract: Many parameter-efficient methods generate the parameters of a large neural network from a low-dimensional latent representation. Given an architecture Phi with P_Phi parameter slots, we write oldsymbol{heta}_f=G(oldsymbol{xi}_f), where GolonR^MoR^{P_Phi} is a parameter generator and oldsymbol{xi}_finR^M is a latent representation of the target function f. The architecture Phi and the generator G are shared across the entire target class, while each target f is represented by its own latent vector oldsymbol{xi}f, with Phi{G(oldsymbol{xi}_f)} approximating f. This framework encompasses hypernetworks, low-dimensional parameterizations, parameter-efficient adaptation, and model compression. Understanding the tradeoff between the latent dimension M and the network budget P is therefore fundamental to characterizing the expressive efficiency of these methods. We study this tradeoff for affine generators and fully connected ReLU architectures. More precisely, optimizing jointly over architectures Phi satisfying P_Phileq P and affine generators G:R^Mo R^{P_Phi}, we prove that the optimal worst-case uniform approximation error over the unit ball of alpha-Holder functions on [0,1]^d, where 0<alphaleq1, has the sharp order igl(Pmin{M,P}igr)^{-alpha/d}. In particular, our result shows that even a fixed-dimensional latent space suffices to achieve vanishing approximation error as the network budget increases.

Related

• Kilobyte Models: Neural Networks as a Seed and a Quantized Latent
• On Explicit Super-Expressive Approximation for Neural Networks
• Histogram-based Parameter-efficient Tuning for Passive and Active Sonar Classification

Source: arXiv cs.LG | 2026-09-01