Three-dimensional scene graph autolabeling

A system comprising one or more processors to:

• obtain, from a simulation environment, one or more views of a three-dimensional (3D) scene for generating a 3D graph of the 3D scene;
• determine a first edge representing an action of a subject on an object in the 3D scene;
• determine, based at least on the action and one or more bounding boxes of the subject in the one or more views of the 3D scene, a caption representing the subject;
• determine, based at least on 3D coordinates for the 3D scene, a second edge representing a spatial relationship between the subject and the object;
• and
• generate the 3D graph of the 3D scene using at least the first edge, the second edge, the caption, a first node corresponding to the subject, and a second node corresponding to the object.

A method, comprising:

• obtaining, from a simulation environment, a dataset comprising at least a plurality of views associated with a three-dimensional (3D) scene;
• and
• updating, based at least on the dataset, a 3D graph of the 3D scene by:
• generating a first node representing a subject in the 3D scene and a second node representing an object in the 3D scene;
• generating a first edge representing an action of the subject with respect to the object;
• generating, based at least on the action and one or more bounding boxes of the subject in the plurality of views of the 3D scene, a caption representing the subject;
• and
• generating, based at least on 3D coordinates for the 3D scene, a second edge representing a spatial relationship between the subject and the object.