Voxel-level feature fusion with graph neural networks and diffusion for 3d object detection

A method for processing image data and point cloud data, the method comprising:

• processing a joint graph representation using a graph neural network (GNN) to form an enhanced graph representation, wherein the joint graph representation includes first features from a voxelized point cloud, and second features from a plurality of camera images, and wherein the enhanced graph representation includes enhanced first features and enhanced second features;
• performing a diffusion processes on the enhanced first features and the enhanced second features of the enhanced graph representation to form a denoised graph representation having denoised first features and denoised second features;
• fusing the denoised first features and the denoised second features of the denoised graph representation using a graph attention network (GAT) to form a fused point cloud having fused features;
• and
• performing a 3D image segmentation process on the fused point cloud.

A non-transitory computer-readable storage medium storing instructions that, when executed, causes one or more processors of a device configured to process image data and point cloud data to:

• process a joint graph representation using a graph neural network (GNN) to form an enhanced graph representation, wherein the joint graph representation includes first features from a voxelized point cloud, and second features from a plurality of camera images, and wherein the enhanced graph representation includes enhanced first features and enhanced second features;
• perform a diffusion processes on the enhanced first features and the enhanced second features of the enhanced graph representation to form a denoised graph representation having denoised first features and denoised second features;
• fuse the denoised first features and the denoised second features of the denoised graph representation using a graph attention network (GAT) to form a fused point cloud having fused features;
• and
• perform a 3D image segmentation process on the fused point cloud.