Language-Guided Object Placement in 3D Scenes

A computer-implemented method comprising:

• receiving an identification of a 3D scene, an identification of a 3D asset, and a natural language prompt including one or more requirements for positioning of the 3D asset in the 3D scene;
• generating a scene vector representing the 3D scene;
• generating an asset vector representing the 3D asset;
• applying the scene vector, the asset vector, and the natural language prompt as input to a large language model (LLM), the LLM generating one or more tokens in response to receiving the natural language prompt;
• applying the one or more tokens as input to a decoder, the decoder generating one or more masks relating to the 3D scene using the one or more tokens;
• and
• identifying one or more recommended positions for the 3D asset in the 3D scene using the one or more masks.

A system comprising:

• one or more processors;
• and
• a non-transitory computer-readable medium comprising instructions, the instructions, when executed by the one or more processors, causing the system to:
• receive an identification of a 3D scene, an identification of a 3D asset, and a natural language prompt including one or more requirements for positioning of the 3D asset in the 3D scene;
• generate a scene vector representing the 3D scene;
• generate an asset vector representing the 3D asset;
• apply the scene vector, the asset vector, and the natural language prompt as input to a large language model (LLM), the LLM generating one or more tokens in response to receiving the natural language prompt;
• apply the one or more tokens as input to a decoder, the decoder generating one or more masks relating to the 3D scene using the one or more tokens;
• and
• identify one or more recommended positions for the 3D asset in the 3D scene using the one or more masks.