RecGen3D: Reconstruction-Guided 3D Generation in a Shared Canonical Space

arXiv:2604.01479v3 Announce Type: replace Abstract: Sparse-view 3D modeling represents a fundamental tension between reconstruction fidelity and generative plausibility. While feed-forward reconstruction excels in efficiency and input alignment, it often lacks the global priors needed for structural completeness. Conversely, diffusion-based generation provides rich geometric details but struggles with multi-view consistency. We present RecGen3D, a framework that combines these two paradigms into a cooperative system. To overcome inherent conflicts in coordinate spaces, 3D representations, and training objectives, we align both models within a shared canonical space. We employ decoupled cooperative learning, which maintains stable training while enabling seamless collaboration during inference. Specifically, the reconstruction module is adapted to provide canonical geometric anchors, while the diffusion generator leverages latent-augmented conditioning to refine and complete the geometric structure. Experimental results demonstrate that RecGen3D achieves superior fidelity and robustness, outperforming existing methods in creating complete and consistent 3D models from sparse observations.

Related

• The Less You Depend, The More You Learn: Synthesizing Novel Views from Sparse, Unposed Images with Minimal 3D Knowledge
• Mix3R: Mixing Feed-forward Reconstruction and Generative 3D Priors for Joint Multi-view Aligned 3D Reconstruction and Pose Estimation
• StructSplat: Generalizable 3D Gaussian Splatting from Uncalibrated Sparse Views

Source: arXiv cs.CV | 2026-08-24