Research
SpaTime: Streaming Vision-Language Models for Spatio-temporal Reasoning
arXiv:2610.08713v1 Announce Type: new Abstract: Embodied agents must reason about 3D space while the video is still arriving, answering questions as soon as they have observed enough of the scene. VLM
arXiv:2610.08713v1 Announce Type: new Abstract: Embodied agents must reason about 3D space while the video is still arriving, answering questions as soon as they have observed enough of the scene. VLMs that incorporate 3D geometric priors achieve strong spatial reasoning, but they operate offline, i.e., the full video must be available before they produce an answer. Streaming VLMs process frames causally and decide for themselves when to respond, yet they lack explicit 3D representations. We present SpaTime, a streaming VLM that fuses causal geometry tokens into the language model at every frame, using only the frames observed so far. To supervise when the model answers, we propose a response-time loss that maps per-frame response probabilities to a differentiable expected response time and penalizes the distance from the ground-truth frame. For evaluation, we construct StreamVSTI-Bench and StreamVSI-Bench, streaming adaptations of VSTI-Bench and VSI-Bench. On StreamVSTI-Bench, SpaTime reaches 49.2% overall accuracy and reduces the mean response-time error by 66% relative to the strongest streaming baseline.
Related
- CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding
- MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering
- CoRDS: Coreset-based Representative and Diverse Selection for Streaming Video Understanding
- Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding
Source: arXiv cs.CV | 2026-10-07