Multi-modal encoder channel fusion with cross-modality awareness

An apparatus, comprising:

• a memory storing processor-readable code;
• and
• at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations including:
• receiving an image frame representing a scene;
• receiving point cloud data representing the scene;
• determining a plurality of first sets of features of the image frame;
• determining a plurality of second sets of features of the point cloud data based on a plurality of voxels representing the point cloud data;
• determining a third set of features of the image frame based on a first set of features of the plurality of first sets of features of the image frame and a second set of features of the plurality of second sets of features of the point cloud data;
• and
• outputting fused data that combines the third set of features of the image frame and a fourth set of features of the point cloud data.

A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

• receiving an image frame representing a scene;
• receiving point cloud data representing the scene;
• determining a plurality of first sets of features of the image frame;
• determining a plurality of second sets of features of the point cloud data based on a plurality of voxels representing the point cloud data;
• determining a third set of features of the image frame based on a first set of features of the plurality of first sets of features of the image frame and a second set of features of the plurality of second sets of features of the point cloud data;
• and
• outputting fused data that combines the third set of features of the image frame and a fourth set of features of the point cloud data.

A vehicle, comprising:

• an image sensor;
• a ranging sensor;
• a processor;
• and
• a memory storing instructions, which when executed by the processor, cause the processor to perform operations including:
• receiving, from the image sensor, an image frame representing a scene;
• receiving, from the ranging sensor, point cloud data representing the scene;
• determining a plurality of first sets of features of the image frame;
• determining a plurality of second sets of features of the point cloud data based on a plurality of voxels representing the point cloud data;
• determining a third set of features of the image frame based on a first set of features of the plurality of first sets of features of the image frame and a second set of features of the plurality of second sets of features of the point cloud data;
• and
• outputting fused data that combines the third set of features of the image frame and a fourth set of features of the point cloud data.