Camera soiling detection using attention-guided camera depth and lidar range consistency gating

An apparatus for camera soiling detection, the apparatus comprising:

• a memory for storing a plurality of images;
• and
• processing circuitry in communication with the memory, wherein the processing circuitry is configured to:
• receive the plurality of images, wherein a first image of the one or more images comprises a range image and a second image comprises a camera image;
• filter the first image to generate a filtered first image, wherein the filtering includes a filter configured to fill in one or more sparse regions in the first image;
• generate a plurality of depth estimates based on the second image;
• generate an attention map by combining the filtered first image and the plurality of depth estimates;
• generate a consistency score indicative of a consistency of depth estimates between the first image and the second image based on the attention map;
• modulate one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features;
• and
• generate a classification of one or more soiled regions in the second image based on the modulated one or more features.

A computer-readable medium storing instructions that, when applied by processing circuitry, causes the processing circuitry to:

• receive a plurality of images, wherein a first image of the one or more images comprises a range image and a second image comprises a camera image;
• filter the first image to generate a filtered first image, wherein the filtering includes a filter configured to fill in one or more sparse regions in the first image;
• generate a plurality of depth estimates based on the second image;
• generate an attention map by combining the filtered first image and the plurality of depth estimates;
• generate a consistency score indicative of a consistency of depth estimates between the first image and the second image based on the attention map;
• modulate one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features;
• and
• generate a classification of one or more soiled regions in the second image based on the modulated one or more features.

A device for camera soiling detection, the device comprising one or more means for performing steps of:

• receiving a plurality of images, wherein a first image of the one or more images comprises a range image and a second image comprises a camera image;
• filtering the first image to generate a filtered first image, wherein the filtering includes a filter configured to fill in one or more sparse regions in the first image;
• generating a plurality of depth estimates based on the second image;
• generating an attention map by combining the filtered first image and the plurality of depth estimates;
• generating a consistency score indicative of a consistency of depth estimates between the first image and the second image based on the attention map;
• modulating one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features;
• and
• generating a classification of one or more soiled regions in the second image based on the modulated one or more features.