Research
Can Learning to Answer Spatial Questions Improve Novel-View Generation?
arXiv:2610.20610v1 Announce Type: new Abstract: Generating novel views from limited observations requires preserving both the appearance and spatial structure of a scene. Although recent approaches ex
arXiv:2610.20610v1 Announce Type: new Abstract: Generating novel views from limited observations requires preserving both the appearance and spatial structure of a scene. Although recent approaches exploit fine-grained geometric cues, they do not explicitly model high-level semantic and spatial properties such as object identity, attributes, and inter-object relationships. In this work, we investigate whether 3D language understanding can provide complementary supervision for novel-view generation. We introduce Ask4NV, built on a pretrained unified multimodal model with a Mixture-of-Transformers (MoT) architecture. Our model incorporates camera geometry into both VAE and ViT representations and leverages understanding features during generation through joint self-attention with a modality-aware attention masking strategy. Experiments demonstrate strong image quality and geometric consistency in novel-view generation, alongside competitive 3D language understanding. Further analysis shows that joint training improves cross-view correspondence in VAE representations and similarity in a 3D-aware feature space.
Related
- Scenix: Sparse-View 3D Scene Reconstruction via Executable Scene Programs
- ToLL: Topological Layout Learning with Asymmetric Cross-View Structural Distillation for 3D Scene Graph Generation Pretraining
- SplatGuide: Geometric Priors from 3D Gaussians for Pose-Free Novel View Synthesis
- UniGeo: Unifying Geometric Guidance for Camera-Controllable Image Editing via Video Models
- MetaView: Monocular Novel View Synthesis with Scale-Aware Implicit Geometry Priors
Source: arXiv cs.CV | 2026-10-15