Research
Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs
arXiv:2602.15846v2 Announce Type: replace Abstract: Decoder-only large language models achieve strong broad performance but are brittle to minor grammatical perturbations, undermining reliability for
arXiv:2602.15846v2 Announce Type: replace Abstract: Decoder-only large language models achieve strong broad performance but are brittle to minor grammatical perturbations, undermining reliability for downstream reasoning. However, directly injecting explicit syntactic structure into an existing checkpoint can interfere with its pretrained competence. We introduce a checkpoint-compatible gated tree cross-attention (GTCA) branch that reads precomputed constituency chunk memory while leaving backbone architecture unchanged. Our design uses a token update mask and staged training to control the scope and timing of structural updates. Across benchmarks and Transformer backbones, GTCA strengthens syntactic robustness beyond continued-training baselines without compromising Multiple-Choice QA performance or commonsense reasoning, providing a practical checkpoint-compatible route to more syntax-robust decoder-only LLMs.
Related
- On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks
- Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy
- Stop Tracking Me! Proactive Defense Against Attribute Inference Attack in LLMs
- Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder
Source: arXiv cs.CL | 2026-04-28