Why Smaller Is Slower? Dimensional Misalignment in Compressed LLMs

arXiv:2604.09595v1 Announce Type: cross Abstract: Post-training compression reduces LLM parameter counts but often produces irregular tensor dimensions that degrade GPU performance -- a phenomenon we call dimensional misalignment. We present a full-stack analysis tracing root causes at three levels: framework, library, and hardware. The key insight is that model inference becomes slower because the resulting dimensions are unfriendly with the GPU execution stack. For example, compressing Llama-3-8B with activation-aware singular value decomposition (ASVD) has 15% fewer parameters yet runs no faster than the uncompressed baseline, because 95% of its dimensions are misaligned. We propose extbf{GAC} (GPU-Aligned Compression), a new compression paradigm that wraps any dimension-reducing compressor and re-selects hardware-aligned dimensions via multi-choice knapsack optimization under the same parameter budget. We evaluate GAC on Llama-3-8B with ASVD and LLM-Pruner, achieving 100% alignment and recovering up to 1.5imes speedup while preserving model quality.

Related

• UCCL-Zip: Lossless Compression Supercharged GPU Communication
• Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
• Constraining Sequential Model Editing with Editing Anchor Compression
• Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration

Source: arXiv cs.AI | 2026-04-14