Research
Sequential Functional Structured Tucker Compression for Large Language Model Attentions
arXiv:2610.00717v1 Announce Type: cross Abstract: Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention
arXiv:2610.00717v1 Announce Type: cross Abstract: Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.
Related
- From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
- Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression
Source: arXiv cs.AI | 2026-10-02