This section shows results on text-conditioned 3D scene generation (left column) and editing-instruction-following (right column), with text inputs shown below the corresponding output renderings. Our representation provides an intuitive interface for controllable scene generation and editing, as 1) program function names correspond to words in natural language, offering interpretable semantic meanings, and 2) program structure reflects scene structure, enabling significant scene changes through simple, one-line edits while preserving overall structure.

The proposed representation captures the structure not only for static, but also for dynamic scenes, and can be applied for synthesizing 4D scenes conditioned on text inputs. Comparisons with a 4D synthesis method, 4D-fy [8], are shown below. Different from 4D-fy which uses and implicit 4D representation, ours explicitly represents the temporal correspondence of entities—Click the button below for tracking visualizations.

The same representation can be rendered with different renderers, showing the versatility of the proposed representation. Different renderers produce renderings that adhere to the same representation and therefore are visually aligned, while each exhibits a different imaging style. The following shows text-conditioned 3D generation results, with the renderer names and input text prompts shown below corresponding renderings.

The proposed representation can be used for image parsing and generating 3D scenes consistent with the parsed image structure and content. In the results below, input images are shown in the first column and renderings with two graphic renderers are shown in the second and third columns.

To automatically infer the representation from text or image inputs, we convert the Scene Language grammar into Python and prompt Claude 3.5 Sonnet [1] to generate programs and words. Neural embeddings reside in the CLIP text embedding space [2,3] and are obtained from texts via the CLIP text encoder or inverted from images with a pre-trained text-to-image model [4,5]. Afterward, a program interpreter executes the program and computes a data object for the scene, and a graphics renderer renders the data object into an image.

We evaluate our method on text-conditioned 3D generation tasks via a user study across 9 scenes shown below. We compare with two baseline methods: GraphDreamer [6], which uses scene graphs for intermediate representation, and MVDream [7], which directly generates scenes from text inputs. Input text prompts are shown below output renderings. Our representation offers flexible and precise specifications for entity relations, which in practice offloads the burden of reasoning about complex entity relations from the graphics renderer towards the language model in our inference pipeline and produces accurate scene generation results.

Failure case 1: prompt sensitivity. For text-conditioned tasks, minor variations in textual scene descriptions can lead to large quality differences in the output.

Failure case 2: image parsing errors. For image-conditioned tasks, input images are parsed with the backbone visual language model (Claude Sonnet 3.5). In the example below, with the same input image, parsing results have high variance across multiple inference runs.