RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
arXiv:2605.15514v1 Announce Type: cross Abstract: We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis a