Code-as-Room: Generating 3D Rooms from Top-Down View Images via Agentic Code Synthesis

Summary (Overview)

• Core Contribution: Code-as-Room (CaR) is a novel Multimodal Large Language Model (MLLM)-based agentic framework that generates executable Blender code to synthesize complete, editable 3D indoor scenes from a single top-down view image.
• Structured Execution: To overcome the instability and infinite looping of existing agents, CaR introduces a structured execution harness that decomposes the complex task into a principled, multi-stage pipeline: scene structuring, layout generation, object profiling, object-level code generation, and interior decoration.
• Cross-Stage Memory: A key innovation is a cross-stage memory module that stores intermediate outputs (e.g., scene graphs, object profiles) and shares them across stages, effectively mitigating the pervasive "context forgetting" problem in long agentic workflows.
• Comprehensive Benchmark: The authors introduce a dedicated benchmark for evaluating code-based 3D room synthesis, assessing models across visual understanding, spatial reasoning, code generation, and holistic scene quality.
• Superior Performance: Extensive experiments show that CaR's harness significantly improves the stability and quality of generation for various MLLM backbones (Gemini, GPT), outperforming direct generation and existing agent-based methods like VIGA in terms of layout alignment, spatial consistency, and scene usability.

Introduction and Theoretical Foundation

Designing realistic 3D indoor rooms is essential for interior design, virtual reality, gaming, and embodied AI. Manual creation is labor-intensive, and while procedural or rule-based methods exist, they are limited by hand-crafted rules. Recent MLLM-based approaches show promise but have key limitations:

• Text-based methods struggle to capture precise spatial information from descriptions.
• Existing image-conditioned agents (e.g., VIGA) suffer from instability and infinite loops when tasked with holistic room generation from top-down views, failing to recover fine-grained details.

The paper proposes a new paradigm: using a top-down layout image as a global spatial prior to guide complete 3D room generation. This mirrors real-world design workflows where floor plans are common starting points. The core idea is to represent the final 3D scene as executable Blender code, which offers interpretability and editability. The challenge is to stably translate a complex visual input into a coherent, multi-faceted code program.

Methodology

The CaR framework is a multi-stage agentic pipeline orchestrated by a structured execution harness. The overall process is formulated as generating executable Blender code $C$ from an input image $I$ via an agent $A$:

$$C = A(I)$$

Due to complexity, the process is decomposed into coarse ($CU$) and fine ($FU$) stages:

$$D_{CU} = U_{CU}(I, M), \quad C_{layout} = G_{CG}(I, D_{CU}, M)$$ $$D_{FU} = U_{FU}(I, C_{layout}, M), \quad C = G_{FG}(I, C_{layout}, D_{FU}, M)$$

where $D$ denotes understanding results, $G$ denotes generators, and $M$ is the cross-stage memory.

1. Cross-Stage Memory System A shared memory $M$ stores typed artifacts $e_s = \langle s, \tau_s, O_s, \eta_s \rangle$ from each stage $s$. Memory is updated via:

$$M_s = M_{s-1} \oplus e_s$$

Downstream stages read predefined views of $M$, ensuring consistency and reducing prompt noise.

2. Image-based Scene Structuring (Stages 1-2)

• Stage 1: Spatial Semantic Analysis. The VLM parses $I$ to extract a schema-constrained description $D_1$ of functional zones, object hierarchies (with identifiers, categories, placement types), and architectural elements (walls, doors, windows).
• Stage 2: Object-centric Scene Graph Construction. Reads $\{I, D_1\}$ to build a deterministic skeleton scene graph $S = \{V_{arch}, V_{major}, E_{parent}, M_{minor}\}$. The VLM completes attributes and forward relations. The final graph $G=(V,E)$ and minor-object sidecar $M_{minor}$ are written to memory.

3. Layout Code Generation (Stages 3-4) Generates coarse layout program $C_{layout}$ where objects are instantiated as bounding-box proxies.

• Stage 3: Major Layout with Visual Feedback. Implements a render–critique–revise loop:
  • Initial code: $C^{(0)} = Generate(I, D_1, G)$
  • Render: $R^{(t)} = Render(C^{(t-1)})$
  • Critique: $(A^{(t)}, s_t) = Critique(I, R^{(t)}, G)$ (VLM provides feedback $A^{(t)}$ and score $s_t$)
  • Sanitize: $eA^{(t)} = Sanitize(A^{(t)}, D_1, G)$
  • Revise: $C^{(t)} = Revise(C^{(t-1)}, eA^{(t)})$ The loop runs for a maximum $T_{max}=5$ iterations or until $s_t \ge s^\star$. Output is $C^{major}_{layout}$.
• Stage 4: Auxiliary Layout. Appends wall-mounted objects and visually salient minor objects $M^{\star}_{minor}$ (e.g., rugs, large plants) to $C^{major}_{layout}$, resulting in the final $C_{layout}$.

4. Object-level Code Generation (Stages 5-7)

• Stage 5: Layout-grounded Object Profiling. Parses $C_{layout}$ and uses the VLM, conditioned on $I$ and memory, to infer fine-grained object descriptions $D_{FU}$ (color, material, function, style).
• Stage 6: Object Geometry Replacement. For each object $o_i$, a geometry agent predicts a semantic 3D primitive decomposition: $$P_i = \Phi_{geo}(o_i, d_i) = \{p_{i,j}\}_{j=1}^{K_i}$$ where $d_i \in D_{FU}$. Proxy constructors in $C_{layout}$ are replaced with part-based constructors to create $C_{geom}$.
• Stage 7: Asset Retrieval for Tiny Objects. For complex small items, simple placeholders are created and then replaced by retrieved assets $b^\star$ from a library $B$, selected via: $$b^\star = \arg\max_{b \in B} \text{match}(b; \text{label}, \text{description}, \text{placeholder size})$$

5. Interior Decoration Code Generation (Stages Corner8-10) A geometry-preserving code rewriting pipeline: $C_{obj} \xrightarrow{ApplyMat} C_{mat} \xrightarrow{ApplyTex} C_{tex} \xrightarrow{RenderSetup} C_{raw}$.

• Stage 8: Material Assignment. Assigns part-level PBR materials based on part dictionary and descriptions.
• Stage 9: Texture and Decorative Surfaces. Uses an image generation model to synthesize texture maps for large surfaces (walls, floors) and injects them.
• Stage 10: Lighting, Rendering, and Post-hoc Correction. VLM infers lighting style, translated into Blender lights. A deterministic correction pass fixes common issues and performs local search for movable objects with boundary violations: $$x_i^\star = \arg\min_{x \in N(\hat{x}_i)} \|x - \hat{x}_i\|_2 \quad \text{s.t.} \quad B(o_i, x) \subseteq B_{room}, \quad B(o_i, x) \cap B(o_j) = \emptyset$$ where $\hat{x}_i$ is the generated position and $N(\hat{x}_i)$ is a local neighborhood. The final program is $C = PostHoc(C_{raw})$.

Empirical Validation / Results

1. Benchmark for Top-down Image to 3D Room A test suite of 41 diverse scenes (residential & specialized, simple to hard, various image styles) was created. Human annotators corrected coarse labels from an initial CaR run to establish annotations. The benchmark evaluates VLMs across four aspects:

Table 1: Benchmark results evaluating different VLMs with and without the Code-as-Room harness.

Key Findings:

• The CaR harness dramatically improves all metrics for VLMs compared to direct single-pass generation.
• Gemini models become the most stable when equipped with CaR, achieving near-perfect agent completion and execution rates.
• GPT-5.5 with CaR achieves the highest object recall but has lower completion/execution rates than Gemini variants.
• Qualitative results (Figs. 3, 4, 5) show CaR produces more complete, structurally sound, and spatially consistent rooms compared to direct generation.

2. Human Evaluation 20 experts rated scenes on similarity, usability, lighting alignment, and acceptability (need for only minor corrections).

Table 2: Human evaluation of overall scene quality.

Key Findings:

• CaR variants consistently outperform baselines in similarity, usability, and acceptability.
• CaR with Gemini3.1-Pro achieves the best scores, indicating its scenes are most aligned with the input and usable with minimal correction.
• While VIGA has good lighting, its lower similarity and usability show weaker layout preservation.

3. Scene Re-rendering The generated 3D scenes (with primitive-based geometry) provide strong structural priors. Using GPT-5.5 for image-level re-rendering of these scenes yields more realistic materials and lighting while preserving the original layout and multi-view consistency (Fig. 7), demonstrating the utility of CaR's output as a prior for visual refinement.

4. Ablation Studies Ablations analyze the memory mechanism and visual feedback loop using Gemini3.1-Pro.

Table 3: Ablation study on different components of Code-as-Room.

Key Findings:

• Removing memory degrades all metrics, especially Layout IoU (-15.2%), showing its critical role in maintaining cross-stage spatial consistency.
• Visual feedback improves performance up to 5 iterations, correcting omissions and errors. Performance degrades at 10 iterations, suggesting layout drift or over-correction.

Theoretical and Practical Implications

• Paradigm Shift: Establishes a robust, image-guided paradigm for 3D scene synthesis, moving beyond text-only or unstable agent-based methods.
• Agentic System Design: Demonstrates the importance of a structured execution harness and persistent memory for complex, multi-step generation tasks, providing a blueprint for building reliable MLLM-based creative agents.
• Code as a 3D Representation: Validates executable code as a powerful, editable, and interpretable representation for complex 3D scenes, bridging high-level vision understanding with low-level graphics implementation.
• Practical Applications: The generated editable Blender scenes are directly usable in interior design, VR/AR content creation, game development, and as structured environments for training and testing embodied AI agents.

Conclusion

Code-as-Room presents an effective MLLM-based agentic framework for generating executable 3D room code from top-down images. Its structured multi-stage pipeline, cross-stage memory, and visual feedback loops address the instability of prior methods. The introduced benchmark and comprehensive experiments validate that the proposed harness significantly enhances generation quality and reliability across different MLLM backbones.

Limitations and Future Work:

1. Currently optimized for top-down views; extending to arbitrary-view inputs would increase general applicability.
2. Procedural code generation has limits for geometrically complex objects; improved asset retrieval or generation is needed for higher fidelity.
3. While re-rendering improves visuals, current video models struggle with long, temporally consistent re-rendering.
4. Future Direction: Exploring video generation models as neural renderers for CaR to produce more realistic and coherent scene visualizations.