Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations

arXiv:2601.03779v2 Announce Type: replace Abstract: We explore intrinsic dimension (ID) of LLM representations as a marker of linguistic complexity. Specifically, we test whether ID differences across model layers reflect well-known complexity contrasts established in (psycho)linguistics: coordination vs. subordination, right-branching vs. center-embedding, and unambiguous vs. ambiguous attachment. Our results on six different LLMs show that these contrasts are consistently reflected in ID differences, with more complex phenomena eliciting higher ID profiles. Notably, ID differences emerge at different points across layers for different contrasts, also reaching their peaks at different stages. Further experiments using representational similarity and layer pruning confirm the trends. We conclude that ID is a useful marker of linguistic complexity in LLMs, that it points to similar linguistic processing steps across disparate LLMs, and that it has the potential to differentiate between different types of complexity.

Related

• Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models
• Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models
• Tracing Relational Knowledge Recall in Large Language Models
• Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning Task

Source: arXiv cs.CL | 2026-04-27