Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative

arXiv:2605.07345v1 Announce Type: new Abstract: Mean-pooled cosine similarity is the default metric for comparing neural representations across languages, modalities, and tasks. We establish that this metric is not length-invariant: under the anisotropy that characterizes modern transformer representations, mean-pooled cosine grows monotonically in sequence length, independent of representational content. Empirically, on HumanEvalPack across four code LLMs, the length ratio alone explains R^2 = 0.52--0.75 of cross-language "Python proximity," while AST depth and shared-token fraction add less than 3% of explained variance beyond length. Substituting Centered Kernel Alignment (CKA) reduces explained variance by 83% and reverses the sign of the length coefficient (eta_{len}: +0.86 o -0.37). The same pattern holds in Mistral-7B on parallel WMT pairs (R^2 = 0.23 EN-FR, R^2 = 0.33 EN-DE for cosine; R^2 < 0.01 for CKA). In CLIP ViT-B/32, mean-pooling reduces the length effect relative to EOS-pooling (R^2: 0.21 o {<}0.01), as predicted by the theory's dependence on anisotropy. We argue that length-invariant metrics such as CKA should be the default for cross-representation comparisons, and that recent claims of cross-lingual representational convergence built on mean-pooled cosine warrant re-examination.

Source: arXiv cs.CL | 2026-05-11