Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs

arXiv:2610.01595v1 Announce Type: cross Abstract: Video Large Language Models (VideoLLMs) receive frames in sequential order and interpret how visual content evolves along the temporal axis, yet temporal reasoning remains a persistent weakness across architectures. Reversing the frame order of a video, a transformation that should invert temporal answers, often leaves the final prediction unchanged. We investigate where this failure originates by defining the temporal divergence vector au_l, the layer-wise representational difference induced by reversing temporal order. Tracking its magnitude across layers reveals a consistent temporal divergence profile where the divergence peaks at intermediate layers and progressively diminishes toward the output. We confirm this peak is specific to temporal reasoning and functionally critical for predictions, establishing that VideoLLMs acquire temporal information at intermediate layers but fail to maintain it to the output. This progressive fading motivates our method, Temporal Activation Injection (TAI), which extracts au_l at the peak of the profile for each input and reinjects it into subsequent layers following the measured decay. TAI requires no training and consistently improves temporal reasoning across three VideoLLMs and four benchmarks with negligible impact on non-temporal tasks. Code is available at https://github.com/Youngwoo-git/Before-It-Fades.

Related

• TTF: Temporal Token Fusion for Efficient Video-Language Model
• AdaCodec: A Predictive Visual Code for Video MLLMs
• Evaluation of MLLM-Agnostic Plug-and-Play Keyframe Selection Methods for Long Video Understanding
• VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understanding
• Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

Source: arXiv cs.AI | 2026-10-02