Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines

arXiv:2604.16734v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have recently demonstrated strong capabilities in understanding and generating responses from diverse visual inputs, including high-resolution images and long video sequences. As these models scale to richer visual representations, inference increasingly relies on storing large numbers of vision tokens in the key-value (KV) cache, making memory consumption a central bottleneck. Existing methods address this issue by identifying redundancy in vision tokens and compressing the cache, but such compression is typically applied only after all inputs are processed, resulting in high peak memory usage during the prefill stage. In this work, we show that MLLMs exhibit inherent structural regularities and representational redundancy that can be exploited to control memory growth throughout inference. Based on this insight, we propose a sequential input-compression mechanism that enforces a fixed memory budget by performing structure-aware key-value cache compression during the prefill process. This approach substantially reduces peak memory usage while maintaining generative performance with only minimal degradation, enabling more practical and memory-efficient multimodal inference.

Related

• EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling
• Geometry-Guided 3D Visual Token Pruning for Video-Language Models
• Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decoding
• Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models
• Does Visual Token Pruning Improve Calibration? An Empirical Study on Confidence in MLLMs

Source: arXiv cs.CV | 2026-04-21