IMPACT: Importance-Aware Activation Space Reconstruction

arXiv:2507.03828v4 Announce Type: replace Abstract: Large language models (LLMs) achieve strong performance across diverse domains but remain difficult to deploy in resource-constrained environments due to their size. Low-rank compression is a common remedy, typically minimizing weight reconstruction error under the assumption that weights are low-rank. However, this assumption often does not hold in LLMs. In contrast, LLM activations exhibit a more pronounced low-rank structure, motivating approaches that minimize activation reconstruction error. This shift alone, however, is not sufficient: different activation dimensions contribute unequally to model performance, and treating them uniformly can lead to accuracy loss. We introduce IMPACT, an importance-aware activation reconstruction framework that links compression to its effect on model performance. IMPACT formulates compression as an optimization problem that integrates activation structure with gradient-based importance, deriving a closed-form solution where reconstruction bases arise from an importance-weighted activation covariance matrix. This yields low-rank compression explicitly optimized for accuracy preservation. Experiments across multiple models and tasks demonstrate that IMPACT achieves up to 55.4% greater model size reduction while maintaining accuracy comparable to or better than state-of-the-art baselines.

Related

• Applied Explainability for Large Language Models: A Comparative Study
• Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systems
• Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure
• Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation
• Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes

Source: arXiv cs.LG | 2026-04-22