Interactive labeling method for 3D dynamic object based on time series data, key frames, and interpolated frames

A device comprising:

• a first coordinate conversion module configured for transforming multi-frame local point clouds in same time series to a same global coordinate system using corresponding poses in the multi-frame local point clouds to obtain global point clouds in the same time series;
• a selection module configured for selecting key frames in the global point clouds for a certain dynamic object sequence according to a trajectory of the dynamic object sequence, and manually labeling key three-dimensional (‘3D’) boxes of the dynamic object in the key frames;
• an interpolation module configured for interpolating other frames based on the key 3D boxes to obtain interpolated 3D boxes, wherein the other frames are frames other than the selected key frames;
• a second coordinate conversion module configured for transforming the key 3D boxes and the interpolated 3D boxes into a local coordinate system through corresponding poses of the key 3D boxes and the interpolated 3D boxes, and obtaining potential point sets of the dynamic object (‘POTENTIAL DYNAMIC OBJECT POINT SETS’) according to a dimension of the key 3D boxes and the interpolated 3D boxes;
• a clustering module configured for clustering point sets in each the interpolated 3D boxes (‘INTERPOLATING POINT SETS’) and key 3D boxes (‘KEY POINT SETS’) respectively based on the POTENTIAL DYNAMIC OBJECT POINT SETS to obtain point sets of the dynamic object in the key 3D boxes and the interpolated 3D boxes of the dynamic object sequence;
• a registration module configured for performing point cloud registration on the INTERPOLATING POINT SETS and two of the KEY POINT SETS that are adjacent to the INTERPOLATING POINT SETS (‘TWO ADJACENT KEY POINT SETS’) to obtain an Rt matrix from the KEY POINT SETS to the INTERPOLATING POINT SETS;
• and
• a 3D box conversion module configured for transforming directions and positions of the key 3D boxes through the Rt matrix, and replacing the interpolated 3D boxes at corresponding positions to obtain 3D prediction boxes;
• wherein the function of performing point cloud registration on the INTERPOLATING POINT SETS and TWO ADJACENT KEY POINT SETS to obtain an Rt matrix from the KEY POINT SETS to the INTERPOLATING POINT SETS comprises:
• computing the Rt matrix to the interpolated 3D boxes by using two adjacent key 3D boxes respectively to obtain two rough registration results;
• using the rough registration results of two adjacent key frames as an initial transformation matrix, and constructing a residual and a Jacobian matrix by using normal information of a nearest point in an interpolating point set of the INTERPOLATING POINT SETS;
• constructing a least square problem based on the Jacobian matrix and minimizing the residual by solving the least square problem;
• updating the transformation matrix according to the solution, and performing a convergence test to finally obtain two fine registration results, wherein the step of performing a convergence test comprises testing whether variation of the updated transformation matrix is smaller than a threshold;
• and
• comparing minimum residual values of the two fine registration results, selecting a fine registration result of the two fine registration results with a smaller residual value as the Rt matrix from the KEY POINT SETS to the INTERPOLATING POINT SETS.

Electronic equipment comprising one or more processors, one or more computer readable storage devices, and program instructions stored on at least one of the one or more computer readable storage devices for execution by at least one of the one or more processors, wherein the program instructions are executable to perform a method in which:

• multi-frame local point clouds are transformed in same time series to a same global coordinate system using corresponding poses in the multi-frame local point clouds to obtain global point clouds in the same time series;
• for a certain dynamic object sequence, key frames in the global point clouds are selected according to a trajectory of the dynamic object sequence, and key three-dimensional (‘3D’) boxes of the dynamic object are manually labeled in the key frames;
• other frames are interpolated based on the key 3D boxes to obtain interpolated 3D boxes wherein the other frames are frames other than the selected key frames;
• the key 3D boxes and the interpolated 3D boxes are transformed into the local coordinate system through corresponding poses of the key 3D boxes and the interpolated 3D boxes, and potential point sets of the dynamic object (‘POTENTIAL DYNAMIC OBJECT POINT SETS’) are obtained according to a dimension of the key 3D boxes and the interpolated 3D boxes;
• point sets in each of the interpolated 3D boxes (‘INTERPOLATING POINT SETS’) and key 3D boxes (‘KEY POINT SETS’) are clustered respectively based on the POTENTIAL DYNAMIC OBJECT POINT SETS to obtain point sets of the dynamic object in the key 3D boxes and the interpolated 3D boxes of the dynamic object sequence;
• point cloud registration is performed on the INTERPOLATING POINT SETS and two of the KEY POINT SETS that are adjacent to the INTERPOLATING POINT SETS (‘TWO ADJACENT KEY POINT SETS’) to obtain an Rt matrix from the KEY POINT SETS to the INTERPOLATING POINT SETS;
• and
• directions and positions of the key 3D boxes are transformed through the Rt matrix, and the interpolated 3D boxes at corresponding positions are replaced to obtain 3D prediction boxes;
• wherein the point cloud registration is performed on the INTERPOLATING POINT SETS and TWO ADJACENT KEY POINT SETS to obtain an Rt matrix from the KEY POINT SETS to the INTERPOLATING POINT SETS by:
• computing the Rt matrix to the interpolated 3D boxes by using two adjacent key 3D boxes respectively to obtain two rough registration results;
• using the rough registration results of two adjacent key frames as an initial transformation matrix, and constructing a residual and a Jacobian matrix by using normal information of a nearest point in an interpolating point set of the INTERPOLATING POINT SETS;
• constructing a least square problem based on the Jacobian matrix and minimizing the residual by solving the least square problem;
• updating the transformation matrix according to the solution, and performing a convergence test to finally obtain two fine registration results wherein the step of performing a convergence test comprises testing whether variation of the updated transformation matrix is smaller than a threshold;
• and
• comparing minimum residual values of the two fine registration results, selecting a fine registration result of the two fine registration results with a smaller residual value as the Rt matrix from the KEY POINT SETS to the INTERPOLATING POINT SETS.