Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m

arXiv:2605.24577v1 Announce Type: cross Abstract: Independently trained transformers compute the same function in residual-stream bases that differ by a uniform random rotation on SO(d_{model}). We call this phenomenon polymorphism: same function, mutually unintelligible interior coordinates. One matrix multiplication per model pair removes it: an orthogonal Procrustes fit on a single batch of activations transfers sparse-autoencoder feature dictionaries and steering vectors between independently trained models, with no retraining. The phenomenon is invisible to the standard SAE universality metric. Decoder-column cosine similarity matches across seeds at 98%, the SAE-universality headline number, while an SAE trained on one seed reconstructs another seed's activations at negative explained variance, worse than predicting the constant mean. The decoder columns align; the encoder reads from a rotated frame. A single Procrustes rotation R restores reconstruction to within 0.025 EV of the within-seed ceiling at every internal site. R is Haar-distributed: |R - I|F matches the random-orthogonal prediction sqrt{2 d{model}} to 0.1% at d_{model} = 512, and a Kolmogorov-Smirnov test of R's eigenvalue spectrum against Haar SO(d_{model}) returns p approx 1.000 pooled and per-pair. Diff-of-means steering vectors transfer in three regimes by alignment with R's invariant subspace: clean when pinned by shared output weights, partial when overlapping the rotated subspace, inverted otherwise. With no shared I/O (Pythia), all three collapse to universally inverted. The same rotation account holds across training checkpoints within a single run. Validated on a 104k-parameter Dyck-3 transformer and nine independently-trained Pythia-70m seeds on The Pile, via a pre-registered four-bar operational framework. Frontier-scale (10B+) replication remains open.

Source: arXiv cs.AI | 2026-05-26