# US 2025/0085407 A1 · Camera soiling detection using attention-guided camera depth and lidar range consistency gating

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0085407 A1
- Document ID: US20250085407A1
- Status: Published application
- Kind: A1
- Application number: 18/464,769
- Filed: 2023-09-11
- Published: 2025-03-13
- Earliest claimed: 2023-09-11
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Varun Ravi Kumar; Senthil Kumar Yogamani; Shivansh Rao
- Web page: https://patentgrep.com/patent/US20250085407A1

## CPC

- G01S 7/497: Means for monitoring or calibrating
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06V 10/34: Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
- G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- G01S 2007/4975: {of sensor obstruction by, e.g. dirt- or ice-coating, e.g. by reflection measurement on front-screen}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30168: Image quality inspection
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method includes receiving a plurality of images, wherein a first image of the one or more images comprises a range image and a second image comprises a camera image and filtering the first image to generate a filtered first image. The method also includes generating a plurality of depth estimates based on the second image and generating an attention map by combining the filtered first image and the plurality of depth estimates. Additionally, the method includes generating a consistency score indicative of a consistency of depth estimates between the first image and the second image based on the attention map, modulating one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features, and generating a classification of one or more soiled regions in the second image based on the modulated one or more features.

## Claims

### Claim 1

A method comprising:
receiving a plurality of images, wherein a first image of the one or more images comprises a range image and a second image comprises a camera image;
filtering, by one or more processors, the first image to generate a filtered first image, wherein the filtering includes a filter configured to fill in one or more sparse regions in the first image;
generating, by the one or more processors, a plurality of depth estimates based on the second image;
generating, by the one or more processors, an attention map by combining the filtered first image and the plurality of depth estimates;
generating, by the one or more processors, a consistency score indicative of a consistency of depth estimates between the first image and the second image based on the attention map;
modulating, by the one or more processors, one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features; and
generating, by the one or more processors, a classification of one or more soiled regions in the second image based on the modulated one or more features.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first image is generated in real time by a Light Detection and Ranging (LIDAR) sensor.

### Claim 3 (depends on 1)

The method of claim 1, further comprising extracting one or more features from the first image prior to the filtering the first image.

### Claim 4 (depends on 1)

The method of claim 1, wherein the filter comprises a morphological filter having a binary mask of pixels that expands one or more regions of the first image, wherein the binary mask is associated with a radius, and wherein the radius of the binary mask determines how far the morphological filter expands the one or more regions of the first image.

### Claim 5 (depends on 1)

The method of claim 1, further comprising generating the plurality of depth estimates using a depth decoder.

### Claim 6 (depends on 1)

The method of claim 1, wherein generating the attention map further comprises dividing the second image into a plurality of layers based on a distance from a camera used to acquire the second image.

### Claim 7 (depends on 6)

The method of claim 6, wherein dividing the second image into the plurality of layers further comprises grouping one or more projected points based on the distance from the camera used to acquire the second image.

### Claim 8 (depends on 7)

The method of claim 7, further comprising calculating the distance from the camera based on a plurality of pixel coordinates in a plane of the second image and a center of the camera in a camera coordinate system.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the consistency score comprises combining the attention map with a generated Structural Similarity Index Matrix (SSIM) score.

### Claim 10 (depends on 9)

The method of claim 9, wherein the SSIM score indicates consistency of the plurality of depth estimates.

### Claim 11 (depends on 9)

The method of claim 9, further comprising identifying the one or more soiled regions based on the consistency score.

### Claim 12 (depends on 1)

The method of claim 1, wherein generating the classification further comprises generating the classification of the one or more soiled regions using a softmax function configured to normalize probability distribution.

### Claim 13 (depends on 1)

The method of claim 1, further comprising operating an Advanced Driver Assistance Systems (ADAS) system based on the classification of the one or more soiled regions.

### Claim 14

An apparatus for camera soiling detection, the apparatus comprising:
a memory for storing a plurality of images; and
processing circuitry in communication with the memory, wherein the processing circuitry is configured to:
receive the plurality of images, wherein a first image of the one or more images comprises a range image and a second image comprises a camera image;
filter the first image to generate a filtered first image, wherein the filtering includes a filter configured to fill in one or more sparse regions in the first image;
generate a plurality of depth estimates based on the second image;
generate an attention map by combining the filtered first image and the plurality of depth estimates;
generate a consistency score indicative of a consistency of depth estimates between the first image and the second image based on the attention map;
modulate one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features; and
generate a classification of one or more soiled regions in the second image based on the modulated one or more features.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the first image is generated in real time by a Light Detection and Ranging (LIDAR) sensor.

### Claim 16 (depends on 14)

The apparatus of claim 14, wherein the processing circuitry is further configured to extract one or more features from the first image prior to the filtering the first image.

### Claim 17 (depends on 14)

The apparatus of claim 14, wherein the filter comprises a morphological filter having a binary mask of pixels that expands one or more regions of the first image, wherein the binary mask is associated with a radius, and wherein the radius of the binary mask determines how far the morphological filter expands the one or more regions of the first image.

### Claim 18 (depends on 14)

The apparatus of claim 14, wherein the processing circuitry is further configured to generate the plurality of depth estimates using a depth decoder.

### Claim 19 (depends on 14)

The apparatus of claim 14, wherein the processing circuitry configured to generate the attention map is further configured to divide the second image into a plurality of layers based on a distance from a camera used to acquire the second image.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the processing circuitry configured to divide the second image into the plurality of layers is further configured to group one or more projected points based on the distance from the camera used to acquire the second image.

### Claim 21 (depends on 20)

The apparatus of claim 20, wherein the processing circuitry is further configured to calculate the distance from the camera based on a plurality of pixel coordinates in a plane of the second image and a center of the camera in a camera coordinate system.

### Claim 22 (depends on 14)

The apparatus of claim 14, wherein the processing circuitry configured to generate the consistency score is further configured to combine the attention map with a generated Structural Similarity Index Matrix (SSIM) score.

### Claim 23 (depends on 22)

The apparatus of claim 22, wherein the SSIM score indicates consistency of the plurality of depth estimates.

### Claim 24 (depends on 22)

The apparatus of claim 22, wherein the processing circuitry is further configured to identify the one or more soiled regions based on the consistency score.

### Claim 25 (depends on 14)

The apparatus of claim 14, wherein the processing circuitry configured to generate the classification is further configured to generate the classification of the one or more soiled regions using a softmax function configured to normalize probability distribution.

### Claim 26 (depends on 14)

The apparatus of claim 14, wherein the processing circuitry is further configured to operate an Advanced Driver Assistance Systems (ADAS) system based on the classification of the one or more soiled regions.

### Claim 27

A computer-readable medium storing instructions that, when applied by processing circuitry, causes the processing circuitry to:
receive a plurality of images, wherein a first image of the one or more images comprises a range image and a second image comprises a camera image;
filter the first image to generate a filtered first image, wherein the filtering includes a filter configured to fill in one or more sparse regions in the first image;
generate a plurality of depth estimates based on the second image;
generate an attention map by combining the filtered first image and the plurality of depth estimates;
generate a consistency score indicative of a consistency of depth estimates between the first image and the second image based on the attention map;
modulate one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features; and
generate a classification of one or more soiled regions in the second image based on the modulated one or more features.

### Claim 28

A device for camera soiling detection, the device comprising one or more means for performing steps of:
receiving a plurality of images, wherein a first image of the one or more images comprises a range image and a second image comprises a camera image;
filtering the first image to generate a filtered first image, wherein the filtering includes a filter configured to fill in one or more sparse regions in the first image;
generating a plurality of depth estimates based on the second image;
generating an attention map by combining the filtered first image and the plurality of depth estimates;
generating a consistency score indicative of a consistency of depth estimates between the first image and the second image based on the attention map;
modulating one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features; and
generating a classification of one or more soiled regions in the second image based on the modulated one or more features.

### Claim 29 (depends on 28)

The device of claim 28, wherein the one or more means comprise one or more processors implemented in circuitry.

### Claim 30 (depends on 29)

The device of claim 29, further comprising a memory to store the plurality of images.
