LCT: Losslessly Compressed Tensors with GPU friendly execution for Training

arXiv:2610.22179v1 Announce Type: new Abstract: Lossless tensor compression could reduce memory usage during training, but training requires tensors to be repeatedly re-encoded. We introduce Losslessly Compressed Tensors (LCT), a GPU execution framework designed for this repeated encode-decode regime. LCT exploits the natural redundancy in floating point data to reduce storage for weights, activations and optimizer states. Highly optimized GPU kernels implement Huffman coding with precomputed codebooks for low compression and decompression overhead, essential for training. Our analysis characterizes LCT's storage costs and adaptation to magnitude shifts. LCT achieves a compression ratio of up to 1.455imes, and is demonstrated in training and finetuning settings. On a 162M-parameter NanoGPT, LCT reduces memory usage by 24.6% for a 7.2% step-time increase. In Qwen3-4B fine-tuning, LCT saves up to 23.8% of peak memory, enabling a 3.5imes larger context length in training on the same GPU.

Source: arXiv cs.LG | 2026-10-16