Multimodal 3D object detection using temporal and structure consistency in voxel feature space

A device for processing media data, the device comprising:

• a memory for storing media data;
• and
• a processing system comprising one or more processors implemented in circuitry, the processing system being configured to:
• form a first voxel representation of a three-dimensional space at a first time using a first image of the three-dimensional space captured by a camera of a moving object having a first pose and a first point cloud of the three-dimensional space captured by a unit of the moving object;
• determine a first set of features for voxels in the first voxel representation, the first set of features representing visual characteristics of the corresponding voxels;
• form a second voxel representation of the three-dimensional space at a second time using a second image of the three-dimensional space captured by the camera of the moving object having a second pose and a second point cloud of the three-dimensional space captured by the unit of the moving object;
• determine a second set of features for voxels in the second voxel representation;
• determine correspondences between the voxels in the first voxel representation and the voxels in the second voxel representation according to similarities between the first set of features and the second set of features;
• and
• determine positions of objects in the three-dimensional space relative to the moving object according to the first pose, the second pose, and the correspondences between the voxels in the first voxel representation and the voxels in the second voxel representation, the objects being represented by the voxels in the first voxel representation and the voxels in the second voxel representation.

A device for processing media data, the device comprising:

• means for forming a first voxel representation of a three-dimensional space at a first time using a first image of the three-dimensional space captured by a camera of a moving object having a first pose and a first point cloud of the three-dimensional space captured by a unit of the moving object;
• means for determining a first set of features for voxels in the first voxel representation, the first set of features representing visual characteristics of the corresponding voxels;
• means for forming a second voxel representation of the three-dimensional space at a second time using a second image of the three-dimensional space captured by the camera of the moving object having a second pose and a second point cloud of the three-dimensional space captured by the unit of the moving object;
• means for determining a second set of features for voxels in the second voxel representation;
• means for determining correspondences between the voxels in the first voxel representation and the voxels in the second voxel representation according to similarities between the first set of features and the second set of features;
• and
• means for determining positions of objects in the three-dimensional space relative to the moving object according to the first pose, the second pose, and the correspondences between the voxels in the first voxel representation and the voxels in the second voxel representation, the objects being represented by the voxels in the first voxel representation and the voxels in the second voxel representation.