Ask Self, Ask Others: Relation Is All You Need

arXiv:2608.20172v1 Announce Type: new Abstract: Attention directly derives normalized information flow from pairwise scores. We introduce Relation, an alternative token-mixing primitive that first organizes pairwise evidence into explicit Self and Exchange relations and derives information flow afterward. This relational organization gives rise to Full Relation, FlashRelation, Linear Relation, Hybrid Relation, and a KV-style Relation Cache. Across matched decoder-only models at approximately 10M, 30M, and 100M parameters, Full Relation achieves lower final validation NLL than MHA at all three scales. In a fixed-context reference benchmark, FlashRelation is 3.60-4.41x faster than the materialized Full Relation implementation. Across scale-matched production workloads, it reaches 76.4-84.9% of PyTorch FlashAttention throughput while executing the Full Relation operator. Hybrid Relation uses 75% Linear Relation layers and achieves strong language-modeling quality. These results support a relation-first view of token mixing: ask Self, ask Others, then let Flow follow Relation.

Related

• Higher-Order Token Interactions via Quantum Attention
• StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k
• HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models

Source: arXiv cs.LG | 2026-08-21