Technique for generating a 3d feature volume for a perception task

A computer-implemented method for outputting a three-dimensional (3D) feature volume of a scene, comprising the following steps:

• receiving, at an input layer of a trained 3D backbone network, first 3D point cloud data of a first scene, wherein the 3D backbone network has been trained by:
• receiving multi-view two dimensional (2D) image data of a scene,
• extracting features at multiple scales of spatial resolution from the received multi-view 2D image data,
• receiving, at an input layer of a 3D backbone network, 3D point cloud data of the scene, wherein the 3D backbone network includes an encoder and a decoder with connections at a plurality of scales of spatial resolution for extracting features from the 3D point cloud data,
• injecting, at multiple layers of the decoder and with corresponding scales of spatial resolution, the extracted features, and
• outputting, at an output layer of the 3D backbone network, a 3D feature volume of the scene based on the received 3D point cloud data and on the injected features;
• outputting, at an output layer of the trained 3D backbone network, a first 3D feature volume of the scene based on the received first 3D point cloud data.

A neural network system for outputting a three-dimensional (3D) feature volume of a scene, comprising:

• a first interface which is configured to receive multi-view two dimensional (2D) image data of a scene;
• a feature extraction module, which is configured to extract features at multiple scales of spatial resolution from the received multi-view 2D image data;
• and
• a 3D backbone network including:
• an input layer, which is configured to receive 3D point cloud data of the scene,
• an encoder and a decoder with connections at a plurality of scales of spatial resolution for extracting features from the 3D point cloud data, wherein multiple layers of the decoder are configured to receive an injection, with corresponding scales of spatial resolution, of the extracted features of the multi-view 2D image data, and
• an output layer, which is configured to output a 3D feature volume of the scene based on the received 3D point cloud data and injected features.

A system for outputting a three-dimensional (3D) feature volume of a scene, comprising:

• at least one optical sensor including at least one camera, which is configured for acquiring multi-view two dimensional (2D) image data of a scene;
• at least one sensor, which is configured to acquire 3D point cloud data of the scene;
• and
• a neural network system including:
• a first interface which is configured to receive the multi-view 2D image data of the scene from the at least one optical sensor,
• a feature extraction module, which is configured to extract features at multiple scales of spatial resolution from the received multi-view 2D image data, and
• a 3D backbone network including:
• an input layer, which is configured to receive the 3D point cloud data of the scene,
• an encoder and a decoder with connections at a plurality of scales of spatial resolution for extracting features from the 3D point cloud data, wherein multiple layers of the decoder are configured to receive an injection, with corresponding scales of spatial resolution, of the extracted features of the multi-view 2D image data, and
• an output layer, which is configured to output a 3D feature volume of the scene based on the received 3D point cloud data and injected features.