GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

arXiv:2603.13418v2 Announce Type: replace Abstract: Structured pruning is widely applied to compress large language models (LLMs), but its performance depends heavily on how neuron importance is estimated. Most existing methods rely on activation statistics from a single calibration set, which introduces calibration bias and degrades downstream cross-task generalization. We observe that neurons exhibit heterogeneous distribution sensitivity, ranging from maintaining relatively stable rankings across calibration datasets to showing substantially larger cross-dataset variation. Ignoring this heterogeneity, existing methods rank all neurons in shared spaces with a uniform scoring source, so calibration-specific neurons dominate the ranking and weakly-activated neurons are scored unreliably. To address this, we propose GPrune-LLM, a structured pruning framework that reduces calibration bias by measuring and exploiting the cross-distribution behavior of neurons for fair comparison. Specifically, we restructure the neuron ranking space into behavior-consistent local spaces, adapt the scoring source where the calibration signal is unreliable, and learn per-module sparsity allocation under a global budget. Experiments across multiple models and downstream tasks show that GPrune-LLM improves the generalization of its base pruning metrics, with gains most pronounced at high sparsity, and reduces dependence on the choice of importance metric.

Source: arXiv cs.LG | 2026-08-04