System and method for 3d object detection by an autonomous vehicle in adverse environmental conditions using multimodal fusion

An autonomy computing system of an autonomous vehicle for object detection by the autonomous vehicle in adverse environmental conditions, the autonomy computing system comprising at least one processor in communication with at least one memory device, and the at least one processor programmed to:

• receive sensor data of an environment in which the autonomous vehicle is operating, the sensor data detected from one or more sensors of a plurality of modalities, the plurality of modalities including a first modality and a second modality, the sensor data including first sensor data from one or more sensors of the first modality and second sensor data from one or more sensors of the second modality, the second sensor data being in a bird's eye view (BEV);
• extract first features in the environment based on the first sensor data and second features in the environment based on the second sensor data;
• fuse, in the BEV, the first features and the second features into first enriched features of the first modality and second enriched features of the second modality by:
• representing the first features in the BEV to derive first BEV features, based on depth information of the first features;
• fusing the first features with the second features corresponding to the first BEV features to derive the first enriched features;
• and
• fusing the second features with the first features corresponding to the second features to derive the second enriched features;
• detect object proposals based on the first enriched features and the second enriched features;
• predict objects in the environment based on the object proposals;
• and
• control operation of the autonomous vehicle based on predicted objects.