Method and apparatus for three-dimensional object perception

An electronic device comprising:

• one or more processors;
• and
• a memory storing instructions configured to cause the one or more processors to:
• receive an input image with respect to a three-dimensional (3D) space, an input point cloud with respect to the 3D space, and an input language with respect to a target object in the 3D space;
• use an encoding model to generate candidate image features of partial areas of the input image, a point cloud feature of the input point cloud, and a linguistic feature of the input language;
• select a target image feature corresponding to the linguistic feature from the candidate image features based on similarity scores of similarities between the candidate image features and the linguistic feature;
• generate a decoding output by executing a multi-modal decoding model based on the target image feature and the point cloud feature;
• and
• detect a 3D bounding box corresponding to the target object by executing an object detection model based on the decoding output.

A vehicle comprising:

• a camera configured to generate an input image with respect to a three-dimensional (3D) space;
• a light detection and ranging (lidar) sensor configured to generate an input point cloud with respect to the 3D space;
• one or more processors configured to:
• receive the input image with respect to the 3D space, the input point cloud with respect to the 3D space, and an input language with respect to a target object in the 3D space;
• use an encoding model to generate candidate image features of partial areas of the input image, a point cloud feature of the input point cloud, and a linguistic feature of the input language;
• select a target image feature corresponding to the linguistic feature from the candidate image features based on similarity scores of similarities between the candidate image features and the linguistic feature;
• generate a decoding output by executing a multi-modal decoding model based on the target image feature and the point cloud feature;
• and
• detect a 3D bounding box corresponding to the target object by executing an object detection model based on the decoding output;
• and
• a control system configured to control the vehicle based on the 3D bounding box.