Directional Contextual Representations for Dependency Relations: Why Cross-Direction Pairing Fails

arXiv:2608.20647v1 Announce Type: new Abstract: Splitting a bidirectional LSTM's contextual representation into a forward-only F_i (strictly a function of tokens 1..i) and a backward-only B_i (strictly a function of tokens i..n) beats either alone and beats a fused self-attention representation for dependency relation-type classification. But a specific, natural extension of this idea -- pairing a token's forward state against a candidate's backward state (``cross-direction'' pairing, F_i vs. B_j) -- consistently underperforms same-direction pairing, and the penalty grows, not shrinks, with token distance, both paired-bootstrap significant. We diagnose why using a frozen-trunk methodology: architectural information leakage between directions is impossible by construction (a single-layer BiLSTM, verified by code inspection); 93% of the same-vs-cross gap survives freezing the trunk and training only fresh heads, ruling out training-co-adaptation as the primary cause; linear regression shows partial representational redundancy between F_i and B_i (R^2{=}0.324 vs. 0.028 for a shuffled control) and a linear probe shows partial anticipatory encoding of upcoming tokens in F_i (36.5% vs. 17.2% majority baseline) -- real effects, but neither alone, nor combined, cleanly explains the full gap. Extended frozen-trunk diagnostics (a positional probe and a distance-decay probe) show directional information is genuinely stored but not exactly positioned, and propagates only a few tokens before decaying to baseline -- consistent with, and mechanistically underneath, the distance-growth finding.

Related

• Symmetry in language statistics shapes the geometry of model representations
• Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMs
• FAAST: Forward-Only Associative Learning via Closed-Form Fast Weights for Test-Time Supervised Adaptation
• On the feasibility of dependency parsing of non-human sequences without a gold standard. Is evaluation possible in other species?

Source: arXiv cs.CL | 2026-08-24