Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations

arXiv:2605.01609v1 Announce Type: new Abstract: We test whether the causal inner product of itet{park2024linear} -- defined by the unembedding covariance Sigma -- enables cross-lingual concept transport. Across 17 models and 4 language pairs, a matched-spectrum randomization test finds that Whitened Causal Alignment is indistinguishable from spectral regularization alone (p = 0.95). However, this failure reveals a broader phenomenon: anti-concentration is observed in residual-stream difference-of-means vectors across five architecture families (p < 10^{-33}) and supported by SAE features (e.g., p = 4.5 imes 10^{-19}) and linear probes on Gemma and Llama. We discover a dual geometry: activation-space concept directions anti-concentrate in the spectral tail, while static unembedding-row contrasts concentrate in high-variance directions (p < 10^{-4}). Split-injection causal interventions support the functional basis on Gemma and Llama (Cohen's d up to 1.80), and POS-tag probing across 8 models shows syntax preferentially encodes in the high-variance subspace in 6 of 8 architectures (p < 0.013), with the Qwen~2.5 family showing a significant reversal consistent with architecture-specific spectral structure. These results suggest transformers may rotate semantic content into spectrally quiet regions during contextualized processing, encoding concepts where they can be manipulated with reduced grammatical disruption.

Source: arXiv cs.LG | 2026-05-05