Let Geometry GUIDE: Layer-wise Unrolling of Geometric Priors in Multimodal LLMs

arXiv:2604.05695v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable progress in 2D visual tasks but still struggle to understand physical space in real-world visual streams. Recently, feed-forward geometric foundation models that implicitly extract geometric priors from visual inputs have offered a promising direction. However, many existing geometry-aware MLLMs extract features from a single deep encoder layer and perform one-shot fusion at the language-model input, potentially overlooking complementary information across geometric encoder layers, limiting access to fine-grained geometric cues, and hindering progressive cross-modal adaptation. To address these limitations, we propose GUIDE (Geometric Unrolling Inside MLLM Early-layers), a progressive framework for integrating geometric priors. GUIDE samples and aligns multi-level features from the geometric encoder and sequentially injects them into the early layers of the MLLM. This design enables the MLLM to continuously access multi-granularity geometric cues and progressively integrate them with visual representations during layer-wise semantic abstraction. GUIDE further introduces a dual context-aware gating mechanism at both the token and layer levels, selectively regulating geometric information to reduce redundant injection and interference with pretrained representations. Extensive experiments on spatial reasoning and 3D scene understanding benchmarks, including VSI-Bench, ScanRefer, and Scan2Cap, validate the effectiveness and cross-task applicability of GUIDE. Our 5B and 9B models achieve average scores of 71.5 and 72.1 on VSI-Bench, respectively.

Source: arXiv cs.CV | 2026-08-11