Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity

arXiv:2605.08811v1 Announce Type: cross Abstract: This paper investigates the learning theory of Transformer networks for regression tasks on the compact Euclidean domain [0,1]^d and d-dimensional compact Riemannian manifolds. We propose a novel constructive approximation framework for Transformers that builds local approximations of the target function and aggregates them into a global approximation via softmax partition of unity. This approach leverages the attention mechanism to achieve spatial localization through affine transformations of the input. The softmax activation plays a crucial role in aggregating local approximations to a global output. From an approximation perspective, we prove that a dense Transformer equipped with only two encoder blocks and standard single-hidden-layer point-wise feed-forward networks can achieve a uniform arepsilon-approximation error for alpha-Holder continuous functions with alpha in (0,1] using O(arepsilon^{-d/alpha}) total parameters. Building upon this approximation guarantee, we establish a near minimax-optimal generalization error bound of order Oig(n^{-frac{2alpha}{2alpha+d}} log nig) for the empirical risk minimizer, where n is the training data size. The Transformer architecture studied in this paper is dense, shallow and wide, and employs softmax activation and sinusoidal positional encodings, closely reflecting practical implementations.

Source: arXiv cs.LG | 2026-05-12