Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization

arXiv:2608.12953v1 Announce Type: new Abstract: Structured pruning is a promising approach for compressing large language models (LLMs), yet existing methods rely heavily on greedy heuristics that produce myopic decisions, and often fail to precisely meet target compression budgets. We present SNIPER, a two-stage structured pruning framework that solves a knapsack optimization over coarse-granularity components to yield conditionally optimal parameter allocations with respect to fixed importance estimates, followed by a fine-grained pruning stage to meet strict budget constraints. We introduce the Compression Ratio Adherence Factor (CRAFT) to quantify budget fidelity, showing that while existing pruners deviate from target compression ratios by up to 33%, SNIPER achieves near-exact adherence with a CRAFT score of 0.98. Evaluations across four diverse architectures over a set of 18 tasks spanning five domains demonstrate SNIPER's consistent improvements in average performance retention and task-level stability over six state-of-the-art pruners. Across all pruning configurations, SNIPER achieves an excellent mean rank of 1.25, indicating its robust cross-architectural generalizability and excellent reliability.

Related

• CausalGate: Causal Importance Distillation for Transformer Module Pruning
• Cascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT
• High-Layer Attention Pruning with Rescaling
• From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models
• Two-Stage Regularization-Based Structured Pruning for LLMs

Source: arXiv cs.CL | 2026-08-14