Model Releases
Spatio-Temporal Grounding of Large Language Models from Perception Streams
arXiv:2604.07592v1 Announce Type: new Abstract: Embodied-AI agents must reason about how objects move and interact in 3-D space over time, yet existing smaller frontier Large Language Models (LLMs) st
arXiv:2604.07592v1 Announce Type: new Abstract: Embodied-AI agents must reason about how objects move and interact in 3-D space over time, yet existing smaller frontier Large Language Models (LLMs) still mis-handle fine-grained spatial relations, metric distances, and temporal orderings. We introduce the general framework Formally Explainable Spatio-Temporal Scenes (FESTS) that injects verifiable spatio-temporal supervision into an LLM by compiling natural-language queries into Spatial Regular Expression (SpRE) -- a language combining regular expression syntax with S4u spatial logic and extended here with universal and existential quantification. The pipeline matches each SpRE against any structured video log and exports aligned (query, frames, match, explanation) tuples, enabling unlimited training data without manual labels. Training a 3-billion-parameter model on 27k such tuples boosts frame-level F1 from 48.5% to 87.5%, matching GPT-4.1 on complex spatio-temporal reasoning while remaining two orders of magnitude smaller, and, hence, enabling spatio-temporal intelligence for Video LLM.
Related
- Vision-Language Navigation for Aerial Robots: Towards the Era of Large Language Models
- Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluation
- Which Way Does Time Flow? A Psychophysics-Grounded Evaluation for Vision-Language Models
- FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios
Source: arXiv cs.RO | 2026-04-10