Systems and methods for automatically annotating multimodal data

A system for automatic annotation of multimodal data comprising:

• an object of interest;
• a mobile data collection rig comprising an image sensor and a LiDAR sensor;
• a plurality of mobile beacons located on the object of interest and on the mobile data collection rig;
• one or more processors;
• and
• memory storing computer program code executable by the one or more processors to cause the system to:
• receive data representing ultrasonic time of flight from the plurality of mobile beacons;
• estimate position data for the object of interest and position data for the mobile data collection rig in a global coordinate frame based on the data representing ultrasonic time of flight from the plurality of mobile beacons;
• compute pose information for the object of interest and pose information for the mobile data collection rig based on the estimated position data of the object of interest and the mobile data collection rig in the global coordinate frame,
• compute pose information for the image sensor and pose information for the LiDAR sensor based on the computed pose information for the mobile data collection rig;
• collect image data from the image sensor and LiDAR data from the LiDAR sensor;
• determine a first location in the image data based on the computed pose information of the image sensor and determine a second location in the LiDAR data based on the computed pose information of the LiDAR sensor;
• and
• automatically annotate the image data with a first set of annotations based on the first location and automatically annotate the LiDAR data with a second set of annotations based on the second location.

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by a system comprising:

• an object of interest,
• a mobile data collection rig comprising an image sensor and a LiDAR sensor,
• a plurality of mobile beacons located on the object of interest and on the mobile data collection rig, and
• one or more processors, cause the system to:
• receive data representing ultrasonic time of flight from the plurality of mobile beacons;
• estimate position data for the object of interest and position data for the mobile data collection rig in a global coordinate frame based on the data representing ultrasonic time of flight from the plurality of mobile beacons;
• compute pose information for the object of interest and pose information for the mobile data collection rig based on the estimated position data of the object of interest and the mobile data collection rig in the global coordinate frame,
• compute pose information for the image sensor and pose information for the LiDAR sensor based on the computed pose information for the mobile data collection rig;
• collect image data from the image sensor and LiDAR data from the LiDAR sensor;
• determine a first location in the image data based on the computed pose information of the image sensor and determine a second location in the LiDAR data based on the computed pose information of the LiDAR sensor;
• and
• automatically annotate the image data with a first set of annotations based on the first location and automatically annotate the LiDAR data with a second set of annotations based on the second location.