Method and system for mixing static scene and live annotations for efficient labeled image dataset collection

A computer system, the system comprising:

• a processor;
• and
• a storage device storing instructions that when executed by the processor cause the processor to perform a method, the method comprising:
• obtaining, by a first recording device based on input from one or more sensors, a three-dimensional (3D) mesh of a scene with a plurality of physical objects;
• marking, by the first recording device while in a first mode, first annotations for a physical object displayed in the 3D mesh, wherein marking while in the first mode comprises a user associated with the first recording device moving around the scene and marking the first annotations in a live view setting by using tools on the first recording device;
• switching from the first mode to a second mode by setting a mode on the first recording device;
• displaying, on the first recording device while in the second mode, the 3D mesh including a first projection indicating a two-dimensional (2D) bounding area corresponding to the marked first annotations;
• marking, by the first recording device while in the second mode, second annotations for the physical object or another physical object displayed in the 3D mesh, wherein marking while in the second mode comprises the user marking the second annotations in a static view setting by using the tools on the first recording device or an input device;
• switching from the second mode to the first mode by setting the mode on the first recording device;
• and
• displaying, on the first recording device while in the first mode, the 3D mesh including a second projection indicating a 2D bounding area corresponding to the marked second annotations.

An apparatus comprising a recording device, which comprises:

• a mesh-obtaining module configured to obtain, based on input from one or more sensors, a three-dimensional (3D) mesh of a scene with a plurality of physical objects;
• an annotation-marking module configured to mark, while in a first mode, first annotations for a physical object displayed in the 3D mesh, wherein marking while in the first mode comprises a user associated with the recording device moving around the scene and marking the first annotations in a live view setting by using tools on the recording device;
• a display-managing module configured to switch from the first mode to a second mode by setting a mode on the recording device;
• wherein the display-managing module is further configured to display, while in the second mode, the 3D mesh including a first projection indicating a two-dimensional (2D) bounding area corresponding to the marked first annotations;
• wherein the annotation-marking module is further configured to mark, while in the second mode, second annotations for the physical object or another physical object displayed in the 3D mesh, wherein marking while in the second mode comprises the user marking the second annotations in a static view setting by using the tools on the recording device or an input device;
• wherein the display-managing module is further configured to switch from the second mode to the first mode by setting the mode on the recording device;
• and
• wherein the displaying-managing module is further configured to display, while in the first mode, the 3D mesh including a second projection indicating a 2D bounding area corresponding to the marked second annotations.