Training for Compositional Sensitivity Reduces Dense Retrieval Generalization

arXiv:2604.16351v1 Announce Type: cross Abstract: Dense retrieval compresses texts into single embeddings ranked by cosine similarity. While efficient for recall, this interface is brittle for identity-level matching: minimal compositional edits (negation, role swaps) flip meaning yet retain high similarity. Motivated by geometric results for unit-sphere cosine spaces (Kang et al., 2025), we test this retrieval-composition tension in text-only retrieval. Across four dual-encoder backbones, adding structure-targeted negatives consistently reduces zero-shot NanoBEIR retrieval (8-9% mean nDCG@10 drop on small backbones; up to 40% on medium ones), while only partially improving pooled-space separation. Treating pooled cosine as a recall interface, we then benchmark verifiers scoring token--token cosine maps. MaxSim (late interaction) excels at reranking but fails to reject structural near-misses, whereas a small Transformer over similarity maps reliably separates near-misses under end-to-end training.

Related

• Working Notes on Late Interaction Dynamics: Analyzing Targeted Behaviors of Late Interaction Models
• Self-Aware Vector Embeddings for Retrieval-Augmented Generation: A Neuroscience-Inspired Framework for Temporal, Confidence-Weighted, and Relational Knowledge
• ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrieval
• CRAFT: Training-Free Cascaded Retrieval for Tabular QA

Source: arXiv cs.CL | 2026-04-21