Model Releases
Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization
arXiv:2607.09967v2 Announce Type: replace-cross Abstract: Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but req
arXiv:2607.09967v2 Announce Type: replace-cross Abstract: Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps. Adaptive optimizers such as Adam normalize updates per coordinate, but update steps remain additive; weights with very different magnitudes receive similarly sized absolute changes, producing very different relative perturbations. We introduce extbf{method} (extbf{methodshort}), a weight reparameterization for neural networks that combines a sign-aware symmetric-exponential pathway with an identity-like linear pathway. The symmetric-exponential pathway is near-linear for small raw weights but increasingly curved at larger magnitudes. Additive updates in logarithmic space map to magnitude-proportional changes in effective weight space. The linear pathway provides a direct route through the transform that we hypothesize stabilizes optimization, while learnable scale, curvature, and offset parameters control balance between pathways and the curvature of the exponential pathway. These components create a curved parameter-space geometry that empirically improves speed of loss descent over standard linear parameterization. We also identify a useful mismatched initialization: raw weights are chosen so a symmetric version of the transform matches Xavier statistics, but training uses an asymmetric forward transform that leaves positive weights at full strength while making negative weights smaller in magnitude; in small-model ablations, this improves early optimization and may act as a form of symmetry breaking. We train transformers on OpenWebText over nine widthimesdepth configurations, methodshort reaches matched validation loss in 1.32--1.49imes fewer training steps, with the largest widths seeing the biggest gains.
Related
- Dynamics Reveals Structure: Challenging the Linear Propagation Assumption
- Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions
- Exponential Approximation Rates and Parameter Efficiency of Learnable Bernstein Activations
- Weight Concentration Regularization for Improving Pruning Robustness Under High Sparsity
Source: arXiv cs.AI | 2026-07-15