Predicting LLM Compression Degradation from Spectral Statistics

arXiv:2604.18085v1 Announce Type: new Abstract: Matrix-level low-rank compression is a promising way to reduce the cost of large language models, but running compression and evaluating the resulting models on language tasks can be prohibitively expensive. Can compression-induced degradation be predicted before committing to this compute? We systematically analyze the Qwen3 and Gemma3 model families across four representative low-rank compression methods: vanilla SVD, two ASVD variants, and SVD-LLM. We find that stable rank and information density, measured in bits per parameter, dominate performance degradation. The interaction term gamma dot ar{rho}_s, defined as compression ratio times stable rank, is a robust predictor of accuracy degradation, achieving leave-one-out cross-validation Pearson correlations of 0.890 for attention layers and 0.839 for MLP layers. We provide theoretical intuition for why this predictor succeeds by connecting it to standard SVD truncation bounds and error composition mechanisms in transformer layers. These findings enable a predict-then-compress workflow: compute gamma dot ar{rho}_s from weights, estimate degradation, and invest compute only in desirable configurations.

Related

• D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximation
• Low-Rank Adaptation Redux for Large Models
• MOONSHOT : A Framework for Multi-Objective Pruning of Vision and Large Language Models

Source: arXiv cs.LG | 2026-04-21