Single frame 4d detection using deep fusion of camera image, imaging radar and lidar point cloud

A method for object detection, the method comprising:

• receiving, by a processor, sensor data indicative of one or more objects for each of a camera subsystem, a LiDAR subsystem, and an imaging RADAR subsystem,
• wherein the sensor data includes camera image data, LiDAR point cloud data and imaging RADAR point cloud data and the sensor data is received simultaneously and within one frame for each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem;
• combining, by the processor, the camera image data, LiDAR point cloud data and imaging RADAR point cloud data to create fused raw data;
• extracting, by the processor, one or more feature representations of the objects from the fused raw data;
• generating, by the processor, fused feature maps from the extracted fused raw data;
• and
• generating, by the processor, object classification, object position, object dimensions, object heading and object velocity from the fused feature maps.

A vehicle control system, comprising:

• a processor;
• and
• a memory coupled with and readable by the processor and storing therein a set of instructions, which when executed by the processor, cause the processor to detect objects in a single frame by:
• receiving sensor data indicative of one or more objects for each of a camera subsystem, a LiDAR subsystem, and an imaging RADAR subsystem;
• wherein the sensor data includes camera image data, LiDAR point cloud data and imaging RADAR point cloud data and the sensor data is received simultaneously and within one frame for each of the camera subsystem, the LiDAR subsystem, and the imaging RADAR subsystem;
• extracting one or more feature representations of the objects from the camera image data, LiDAR point cloud data and imaging RADAR point cloud data;
• generating image feature maps from the extracted camera image data, LiDAR feature maps from the LiDAR point cloud data and imaging RADAR feature maps from the image RADAR point cloud data;
• combining the image feature maps, the LiDAR feature maps and the imaging RADAR feature maps to generate merged feature maps;
• and
• generating object classification, object position, object dimensions, object heading and object velocity from the merged feature maps.