Method and device for learning knowledge distillation based on multi-sensor, method and device for inferring image bev feature information, and computer-readable storage medium storing instructions for performing method for learning knowledge distillation based on multi-sensor

An apparatus for a vehicle, the apparatus comprising:

• a plurality of sensors comprising at least one of a camera, a light detection and ranging (LIDAR) sensor, a radar sensor, or an ultrasonic sensor;
• a processor;
• and
• a memory storing at least one instruction that, when executed by the processor, is configured to cause the apparatus to:
• obtain, via the plurality of sensors, sensor data,
• generate, based on the sensor data, at least one piece of sensor data-based voxel feature information,
• perform a view transformation on the at least one piece of sensor data-based voxel feature information to construct sensor data-based view-transformed feature information,
• generate, based on the sensor data-based view-transformed feature information, first voxel bird's-eye-view (BEV) feature information,
• generate, based on the at least one piece of sensor data-based voxel feature information, second voxel BEV feature information,
• generate, based on the first voxel BEV feature information and the second voxel BEV feature information, a fused voxel BEV feature information,
• determine, based on image data obtained via the camera, image feature information,
• train, based on the sensor data-based view-transformed feature information and the image feature information, an image view transformation model, wherein the image view transformation model is trained to perform a view transformation on the image feature information to generate view-transformed image feature information,
• train, based on the fused voxel BEV feature information, an image BEV generation model, wherein the image BEV generation model is trained to generate image BEV feature information corresponding to the view-transformed image feature information,
• transmit, to the vehicle, a signal indicating the image BEV feature information, and
• cause, based on the transmission of the signal, a control operation of the vehicle.

A non-transitory computer-readable medium storing instructions that, when executed, cause an apparatus for a vehicle to:

• obtain, via a plurality of sensors of the vehicle, sensor data generate, based on the sensor data, at least one piece of sensor data-based voxel feature information,
• perform a view transformation on the at least one piece of sensor data-based voxel feature information to construct sensor data-based view-transformed feature information,
• generate, based on the sensor data-based view-transformed feature information, first voxel bird's-eye-view (BEV) feature information,
• generate, based on the at least one piece of sensor data-based voxel feature information, second voxel BEV feature information,
• generate, based on the first voxel BEV feature information and the second voxel BEV feature information, fused voxel BEV feature information,
• determine, based on image data obtained via a camera of the vehicle, image feature information
• train, based on the fused voxel BEV feature information and image feature information, an image view transformation model, wherein the image view transformation model is trained to perform view transformation on the image feature information to generate view-transformed image feature information,
• train, based on the fused voxel BEV feature information, an image BEV generation model, wherein the image BEV generation model is trained to generate image BEV feature information corresponding to the view-transformed image feature information,
• transmit, to the vehicle, a signal indicating the image BEV feature information;
• and
• cause, based on the transmission of the signal, autonomous driving control of the vehicle.