MechSparse: Mechanism-Guided Sparse PEFT Selection Is Task-Shaped

arXiv:2609.18961v1 Announce Type: new Abstract: Mechanistic interpretability identifies sparse subsets of heads and MLP blocks that carry specific behaviors. We ask whether such causal signals can guide where to place a small PEFT budget more effectively than the cheap heuristics practitioners already use. method{} scores attention heads and MLP blocks by normalized activation-patching recovery on clean/corrupted probes and trains LoRA/QLoRA only on the selected sites; methodc{} adds bounded credit for small within-layer joint subsets. We compare against random, magnitude, activation-norm, and gradient/Fisher on Ministral-8B/NF4 in three cells: Swahili span-JSON information extraction (IE) at b{=}0.25% and 1.0%, and EnglishoSwahili machine translation (MT) at b{=}1.0%. The causal selectors never win the primary metric. On the headline IE cell (3 seeds, paired-bootstrap CIs over 600 predictions), methodc{} beats random by +0.079 span+type F1 and gradient/Fisher by +0.174, but trails activation-norm by 0.028, with the smallest cross-seed std (pm 0.003). On MT all four selectors lie within 0.30 BLEU and every paired CI includes zero. A schema-versus-span decomposition explains the IE gap: activation-norm captures the rigid JSON routine, while causal scores track content-sensitive sites. We distill a preliminary diagnostic -- prefer activation-norm when output structure dominates, treat causal selectors as a hypothesis for content-dominated tasks -- and release masks, scores, predictions, and evaluation files for direct replay.

Related

• Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoning
• Sparse Weight Decomposition for Efficient Circuit Extraction
• Targeting the Attention Heads Behind Object Hallucination in LLaVA

Source: arXiv cs.CL | 2026-09-17