LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding

arXiv:2608.25729v1 Announce Type: new Abstract: Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.

Related

• One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding
• WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensation
• EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs
• Ground, Cover, and Refine: Evidence-Centric Frame Selection for Long-Video Question Answering
• QCA: Query- and Content-Aware Keyframe Selection for Long Video Understanding

Source: arXiv cs.CV | 2026-08-27