Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs

arXiv:2610.05417v1 Announce Type: new Abstract: Recent works augment Vision-Language Models with geometry features from pretrained 3D models, expecting that the geometric signal will boost spatial reasoning. However, we find that simply fusing geometry features and training on standard spatial QA yields only marginal improvements on high-level multi-hop tasks. We attribute this gap to a training-signal problem: standard spatial QA can be largely answered from visual features and language priors, so the geometry pathway receives weak gradients and fails to integrate with the visual features. To provide a training signal that requires geometry, we propose extbf{novel-view semantic rendering} as an auxiliary training task that requires the model to predict the semantic layout of an unobserved viewpoint, inspired by humans' ability to mentally simulate novel viewpoints during spatial reasoning. This task encourages joint use of both pathways: geometry provides pose-dependent visibility, while vision provides semantic content. Our auxiliary task yields consistent improvements over the geometry-augmented baseline across all three benchmarks (up to +1.6 on VSI-Bench, +2.2 on ReVSI, +2.9 on our 3D-Point-QA dataset) and our full model surpasses prior open-source methods on VSI-Bench and on ReVSI. Project page: https://yuqunw.github.io/Render2Reason/.

Related

• SpatialForge: Bootstrapping 3D-Aware Spatial Reasoning from Open-World 2D Images
• World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning
• FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views
• Geometric Encoding for Spatial Reasoning in Vision-Language Models

Source: arXiv cs.CV | 2026-10-06