Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusion

arXiv:2604.16552v1 Announce Type: new Abstract: Recent text-to-scene generation approaches largely reduced the manual efforts required to create 3D scenes. However, their focus is either to generate a scene layout or to generate objects, and few generate both. The generated scene layout is often simple even with LLM's help. Moreover, the generated scene is often inconsistent with the text input that contains non-trivial descriptions of the shape, appearance, and spatial arrangement of the objects. We present a new paradigm of sequential text-to-scene generation and propose a novel generative model for interactive scene creation. At the core is a 3D Autoregressive Diffusion model 3D-ARD+, which unifies the autoregressive generation over a multimodal token sequence and diffusion generation of next-object 3D latents. To generate the next object, the model uses one autoregressive step to generate the coarse-grained 3D latents in the scene space, conditioned on both the current seen text instructions and already synthesized 3D scene. It then uses a second step to generate the 3D latents in the smaller object space, which can be decoded into fine-grained object geometry and appearance. We curate a large dataset of 230K indoor scenes with paired text instructions for training. We evaluate 7B 3D-ARD+, on challenging scenes, and showcase the model can generate and place objects following non-trivial spatial layout and semantics prescribed by the text instructions.

Related

• HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models
• UniMesh: Unifying 3D Mesh Understanding and Generation
• SIC3D: Style Image Conditioned Text-to-3D Gaussian Splatting Generation
• Geometrically Consistent Multi-View Scene Generation from Freehand Sketches
• Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale

Source: arXiv cs.CV | 2026-04-21