Research
Do Syntax Heads Influence Model Behavior?
arXiv:2610.19231v1 Announce Type: new Abstract: Transformer language models develop attention heads that specialize in syntactic relations, for example, attending from verbs to their subjects. Do thes
arXiv:2610.19231v1 Announce Type: new Abstract: Transformer language models develop attention heads that specialize in syntactic relations, for example, attending from verbs to their subjects. Do these heads play a causal role in the model's syntactic predictions? Evidence for such a role is mixed to negative: even when we force these heads to attend to a distractor noun other than the subject, the model's syntactic predictions are still more consistent with the subject than with the distractor. Interventions on specialized syntax heads had a significantly stronger effect on syntactic predictions than interventions on random heads, but only in one of the two measures we propose.
Related
- Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time
- Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs
- Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMs
Source: arXiv cs.CL | 2026-10-15