Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

arXiv:2610.19121v1 Announce Type: cross Abstract: Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose extit{LayerTrim}, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth. LayerTrim is the first training-free structured pruning method designed to preserve long-form reasoning without gradients or recovery training. Across three reasoning language models and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On exttt{Qwen3-14B}, LayerTrim achieves 1.24imes faster prefill and 19.7% lower peak GPU memory than the dense model, while better preserving reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation.

Related

• SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference
• Omni-Prune: Query-Aware Unified Token Pruning for Efficient Omnimodal Large Language Models
• BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

Source: arXiv cs.CL | 2026-10-15