Tracing Relational Knowledge Recall in Large Language Models

arXiv:2604.19934v1 Announce Type: new Abstract: We study how large language models recall relational knowledge during text generation, with a focus on identifying latent representations suitable for relation classification via linear probes. Prior work shows how attention heads and MLPs interact to resolve subject, predicate, and object, but it remains unclear which representations support faithful linear relation classification and why some relation types are easier to capture linearly than others. We systematically evaluate different latent representations derived from attention head and MLP contributions, showing that per-head attention contributions to the residual stream are comparatively strong features for linear relation classification. Feature attribution analyses of the trained probes, as well as characteristics of the different relation types, reveal clear correlations between probe accuracy and relation specificity, entity connectedness, and how distributed the signal on which the probe relies is across attention heads. Finally, we show how token-level feature attribution of probe predictions can be used to reveal probe behavior in further detail.

Related

• OmniTrace: A Unified Framework for Generation-Time Attribution in Omni-Modal LLMs
• CoG: Controllable Graph Reasoning via Relational Blueprints and Failure-Aware Refinement over Knowledge Graphs
• How Retrieved Context Shapes Internal Representations in RAG
• Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models

Source: arXiv cs.CL | 2026-04-23