A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization

arXiv:2605.18933v1 Announce Type: new Abstract: Pre-norm Transformers with RMSNorm tolerate ternary {-1,0,+1} weight quantization with surprisingly small loss (Ma et al., 2024). We give a geometric explanation via sign-magnitude decomposition of weight perturbations. In a two-layer ReLU + RMSNorm model with i.i.d. Gaussian weights, sign-flips produce pi/(pi-2) approx 2.75 times more transverse output energy than sign-preserving magnitude perturbations of equal Frobenius norm, as the flip rate p o 0 (Theorem 3). The mechanism: ReLU creates a hidden-space directional asymmetry between the two perturbation types, which RMSNorm's transverse-projection Frechet derivative selectively exposes. Sign-quantization error is itself a sign-preserving perturbation with angular alignment os^2 o 2/pi (Theorem 4); its post-ReLU radial fraction (0.365) matches the pre-ReLU value 1-2/pi within 0.4%, so ReLU is approximately transparent to ternary error. Multi-layer compounding of the 2.75imes factor is not experimentally supported; the gap to real-model sign sensitivity arises from outlier features violating delocalization. For an input dimension with amplitude alpha, a single sign-flip produces post-ReLU energy amplified by R approx nalpha^2 relative to a delocalized entry. On TinyLlama-1.1B, at linear response (p leq 0.5%), count-matched NLL leverage stabilizes at sim 10imes approx nE[alpha^2], matching the per-entry theory; the all-column NLL ratio of 5.0imes falls within R_{col} leq 19 (67imes PPL gap reflects metric nonlinearity). Measured outlier alpha at layer 12 (median 0.024, max 0.26) confirms heavy-tailed concentration. The Bussgang constant 2/pi, RMSNorm geometry, and ReLU half-space structure together explain sign-magnitude asymmetry in pre-norm models, with R propto nalpha^2 accounting for real-model deviations.

Source: arXiv cs.LG | 2026-05-20