Early fusion of camera and radar frames

A method of performing early fusion of camera and radar frames to perform object detection in one or more spatial domains performed by an on-board computer of a host vehicle, comprising:

• receiving, from a camera sensor of the host vehicle, a plurality of camera frames;
• receiving, from a radar sensor of the host vehicle, a plurality of radar frames;
• applying an encoder-decoder network on the first camera frame to generate a first camera feature map in a spatial domain of a first radar frame, wherein the first radar frame corresponds in time to the first camera frame;
• combining the first radar frame and the first camera feature map to generate a first combined feature map in the spatial domain of the first radar frame;
• performing object detection on the first combined feature map to detect one or more objects in the first combined feature map without performing object detection on the first camera feature map or the first radar frame;
• and
• estimating a width, length, or both of the one or more objects in a bird's eye view, after inverse perspective mapping, based on a bounding box in the first camera frame encapsulating each of the one or more object.

An on-board computer of a host vehicle, comprising:

• at least one processor configured to:
• receive, from a camera sensor of the host vehicle, a plurality of camera frames;
• receive, from a radar sensor of the host vehicle, a plurality of radar frames;
• perform a camera feature extraction process on a first camera frame of the plurality of camera frames to generate a first camera feature map;
• perform a radar feature extraction process on a first radar frame of the plurality of radar frames to generate a first radar feature map, wherein the first radar frame corresponds in time to the first camera frame;
• convert the first camera feature map, the first radar feature map, or both to a common spatial domain;
• concatenate the first radar feature map and the first camera feature map to generate a first concatenated feature map in the common spatial domain;
• perform object detection on the first concatenated feature map to detect one or more objects in the first concatenated feature map without performance of object detection on the first camera feature map or the first radar feature map;
• and
• estimate a width, length, or both of the one or more objects in a bird's eye view, after inverse perspective mapping, based on a bounding box in the first camera frame encapsulating each of the one or more objects.

An on-board computer of a host vehicle, comprising:

• at least one processor configured to:
• receive, from a camera sensor of the host vehicle, a plurality of camera frames;
• receive, from a radar sensor of the host vehicle, a plurality of radar frames;
• apply an encoder-decoder network on the first camera frame to generate a first camera feature map in a spatial domain of a first radar frame, wherein the first radar frame corresponds in time to the first camera frame;
• combine the first radar frame and the first camera feature map to generate a first combined feature map in the spatial domain of the first radar frame;
• and
• perform object detection on the first combined feature map to detect one or more objects in the first combined feature map without performance of object detection on the first camera feature map or the first radar frame;
• and
• estimate a width, length, or both of the one or more objects in a bird's eye view, after inverse perspective mapping, based on a bounding box in the first camera frame encapsulating each of the one or more object.