Image and lidar adaptive transformer for fusion-based perception

A method comprising:

• applying, based on a positional encoding model, a first feature conditioning module to a set of bird's eye view (BEV) position data features corresponding to position data to generate a set of conditioned BEV position data features;
• applying, based on the position encoding model, a second feature conditioning module to a set of perspective image data features corresponding to image data to generate a set of conditioned perspective image data features;
• generating, based on the positional encoding model, the set of conditioned BEV position data features, and the set of conditioned perspective image data features, a weighted summation indicating a first relative importance of image data for indicating characteristics of a three-dimensional (3D) environment and a second relative importance of position data for indicating characteristics of the 3D environment;
• and
• generating, based on the weighted summation, a set of BEV image data features.

A computer-readable medium storing instructions that, when applied by processing circuitry, causes the processing circuitry to:

• apply, based on a positional encoding model, a first feature conditioning module to a set of bird's eye view (BEV) position data features corresponding to position data to generate a set of conditioned BEV position data features;
• apply, based on the position encoding model, a second feature conditioning module to a set of perspective image data features corresponding to image data to generate a set of conditioned perspective image data features;
• generate, based on the positional encoding model, the set of conditioned BEV position data features, and the set of conditioned perspective image data features, a weighted summation indicating a first relative importance of the image data for indicating characteristics of a three-dimensional (3D) environment and a second relative importance of the position data for indicating characteristics of the 3D environment;
• and
• generate, based on the weighted summation, a set of BEV image data features.