SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks

arXiv:2606.02927v1 Announce Type: new Abstract: Normalization layers such as BatchNorm and LayerNorm have long been considered essential for stable training in deep networks. This work demonstrates that they can be fully replaced by a single learnable activation mechanism. We identify a plasticity suppression effect induced by standard normalization: learnable activation parameters rapidly lose adaptability when paired with normalization layers. Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), [ operatorname{SALU}(x;a,b) = frac{a x}{sqrt{1 + a b x^2}},quad a>0,; b>0 ] a bounded, learnable activation that provides intrinsic signal stabilization without relying on batch statistics or external affine parameters. Building on SALU, we propose SaluNet, a paradigm grounded in total plasticity: SALU replaces normalization layers, while SWALU and GALU replace standard activations. With ResNet-18, SaluNet-C-18 achieves 97.35% on CIFAR-10 and 83.25% on CIFAR-100 without normalization, maintaining 93.44% and 76.23% at batch size 1 where normalized architectures fail. For transformers, SaluNet-T improves over LayerNorm-GELU from 90.92% to 91.01% on CIFAR-10 and from 66.54% to 68.10% on CIFAR-100. SaluNet-C-50 reaches 78.67% Top-1 on ImageNet-1K at 224imes224, and 79.23% at 288imes288. These results suggest normalization layers suppress total plasticity, a property biological neurons inherently possess, enabling deep networks to learn effectively.

Source: arXiv cs.CV | 2026-06-03