# US 10,769,454 B2 · Camera blockage detection for autonomous driving systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,454 B2
- Document ID: US10769454B2
- Status: Granted
- Kind: B2
- Application number: 16/181,366
- Filed: 2018-11-06
- Published: 2019-05-09
- Granted: 2020-09-08
- Earliest claimed: 2017-11-07
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Xiaoyan Mu; Xiaohan Hu
- Web page: https://patentgrep.com/patent/US10769454B2

## CPC

- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G05D 1/0246: {using a video camera in combination with image processing means}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

System and methods for detecting blockages in images are described. A method may include receiving a plurality of images captured by a camera installed on a vehicle. The method may include identifying one or more candidate blocked regions in the plurality of images. Each of the candidate blocked regions may contain image data caused by blockages in the camera's field-of-view. The method may further include assigning blockage scores to the one or more candidate blocked regions based on region-associations among the one or more candidate blocked regions in the plurality of images. In response to a determination that one of the blockage scores is above a predetermined blockage threshold, the method may include transmitting a blockage alarm signal to the vehicle.

## Claims

### Claim 1

A method for detecting blockages in images, the method comprising:
receiving, by a blockage detection module, a plurality of images captured by a camera installed on a vehicle;
identifying, by the blockage detection module, one or more candidate blocked regions in the plurality of images, wherein each of the candidate blocked regions contains image data caused by blockages in the camera's field-of-view;
assigning, by the blockage detection module, blockage scores to the one or more candidate blocked regions based on region-associations among the one or more candidate blocked regions in the plurality of images; and
in response to a determination that one of the blockage scores is above a predetermined blockage threshold, transmitting, by the blockage detection module, a blockage alarm signal to the vehicle.

### Claim 2 (depends on 1)

The method as recited in claim 1, wherein the identifying of the one or more candidate blocked regions comprises:
dividing an image selected from the plurality of images into a plurality of processing units (PUs);
identifying a plurality of blocked PUs from the plurality of PUs based on a subset of prior images in the plurality of images that are captured prior to the image; and
clustering the plurality of blocked PUs into the one or more candidate blocked regions.

### Claim 3 (depends on 2)

The method as recited in the claim 2, wherein the identifying of the plurality of blocked PUs comprises:
generating a feature value image based on feature values extracted from the plurality of PUs;
generating a feature value threshold based on a histogram of the feature value image;
for a PU in the plurality of PUs, in response to a determination that the PU's corresponding feature value is above the feature value threshold, identifying the PU as an unblocked PU; and
identifying the plurality of blocked PUs based on the unblocked PUs identified in the plurality of PUs.

### Claim 4 (depends on 2)

The method as recited in the claim 2, wherein the identifying of the plurality of blocked PUs comprises:
generating a codebook based on the set of prior images, wherein the codebook includes a codebook box containing minimum feature values extracted from the set of prior images; and
for a specific PU selected from the plurality of PUs, in response to a determination that the specific PU's feature value is within a range defined by the codebook box, identifying the specific PU as one of the blocked PUs.

### Claim 5 (depends on 2)

The method as recited in the claim 2, wherein the identifying of the plurality of blocked PUs comprises:
generating a road-surface waveform based on feature values associated with unblocked road surfaces;
for a specific PU selected from the plurality of PUs, in response to a determination that the specific PU's feature value is within a predetermined range of the road-surface waveform, identifying the specific PU as one of the unblocked PUs; and
identifying the plurality of blocked PUs based on unblocked PUs in the plurality of PUs.

### Claim 6 (depends on 2)

The method as recited in the claim 2, wherein the identifying of the plurality of blocked PUs comprises:
for a specific PU selected from the plurality of PUs, in response to a determination that the specific PU's feature value is within a predetermined range of a reference value generated based on a previously identified unblocked PU, identifying the specific PU as an unblocked PU; and
identifying the plurality of blocked PUs based on unblocked PUs in the plurality of PUs.

### Claim 7 (depends on 2)

The method as recited in the claim 2, wherein the identifying of the plurality of blocked PUs comprises:
maintaining a blockage unit map based on the set of prior images, wherein the blockage unit map contains a plurality of entries corresponding to the plurality of PUs;
adjusting the plurality of entries based on a plurality of unblocked PUs identified from the plurality of PUs; and
generating the plurality of blocked PUs based on the plurality of entries having corresponding counter values that are below a predetermined counter threshold.

### Claim 8 (depends on 2)

The method as recited in the claim 2, wherein the clustering of the plurality of blocked PUs comprises:
utilizing a connected component labeling (CCL) process to cluster the plurality of blocked PUs into the one or more candidate blocked regions.

### Claim 9 (depends on 1)

The method as recited in the claim 1, wherein the assigning the blockage scores to the candidate blocked regions comprises:
generating a blockage region map based on the candidate blocked regions, wherein the blockage region map contains a plurality of region trackers each of which corresponds to one of the candidate blocked regions and is associated with a blockage score;
identifying, using the plurality of region trackers, the region-associations among the candidate blocked regions based on their respective spatial proximity, size, and time in the plurality of images; and
adjusting the blockage scores in the plurality of region trackers based on the region-associations among the candidate blocked regions.

### Claim 10

A non-transitory computer-readable storage medium, containing a set of instructions which, in response to execution by a processor, cause the processor to perform a method for detecting blockages in images, the method comprising:
receiving a plurality of images captured by a camera installed on a vehicle;
for an image selected from the plurality of images,
dividing the image into a plurality of processing units (PUs),
identifying a plurality of blocked PUs from the plurality of PUs based on a subset of prior images in the plurality of images that are captured prior to the image, and
clustering the plurality of blocked PUs into one or more candidate blocked regions;
generating a blockage region map based on the candidate blocked regions extracted from the plurality of images; and
in response to a determination that a corresponding blockage score in the blockage region map is above a predetermined confidence score, transmitting a blockage alarm signal to the vehicle.

### Claim 11 (depends on 10)

The non-transitory computer-readable storage medium as recited in the claim 10, wherein the identifying of the plurality of blocked PUs comprises:
generating a feature value image based on feature values extracted from the plurality of PUs;
generating a feature value threshold based on a histogram of the feature value image;
for a PU in the plurality of PUs, in response to a determination that the PU's corresponding feature value is above the feature value threshold, identifying the PU as an unblocked PU; and
identifying the plurality of blocked PUs based on unblocked PUs in the plurality of PUs.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium as recited in the claim 11, wherein the identifying of the plurality of blocked PUs comprises:
generating a codebook based on the set of prior images, wherein the codebook includes a codebook box containing minimum feature values extracted from the set of prior images; and
for a specific PU that is not one of the plurality of blocked PUs, in response to a determination that the specific PU's feature value is within the codebook box's range, identifying the specific PU as one of the blocked PU.

### Claim 13 (depends on 11)

The non-transitory computer-readable storage medium as recited in the claim 11, wherein the identifying of the plurality of blocked PUs comprises:
generating a road-surface waveform based on feature values associated with unblocked road surfaces;
for a specific PU selected from the plurality of blocked PUs, in response to a determination that the specific PU's feature value is within a predetermined range of the road-surface waveform, identifying the specific PU as an unblocked PU; and
identifying the plurality of blocked PUs based on unblocked PUs in the plurality of PUs.

### Claim 14 (depends on 11)

The non-transitory computer-readable storage medium as recited in the claim 11, wherein the identifying of the plurality of blocked PUs comprises:
for a specific PU selected from the plurality of blocked PUs, in response to a determination that the specific PU's feature value is within a predetermined range of a reference value generated based on a previously identified unblocked PU, identifying the specific PU as an unblocked PU; and
identifying the plurality of blocked PUs based on unblocked PUs in the plurality of PUs.

### Claim 15 (depends on 10)

The non-transitory computer-readable storage medium as recited in the claim 10, wherein the blockage region map contains a plurality of region trackers each of which corresponds to one of the candidate blocked regions and is associated with a blockage score, and the generating of the blockage region map comprises:
identifying, using the plurality of region trackers, region-associations among the candidate blocked regions based on their respective spatial proximity, size, and time in the plurality of images; and
adjusting the blockage scores in the plurality of region trackers based on the region-associations among the candidate blocked regions.

### Claim 16

A system for detecting blockages in images, the system comprising:
a camera installed on a vehicle for capturing a plurality of images;
an alarm installed on the vehicle; and
an autonomous driving system coupled with the camera and the alarm, wherein the autonomous driving system is configured to
identify one or more candidate blocked regions in the plurality of images, wherein each of the candidate blocked regions contains image data caused by blockages in the camera's field-of-view,
assign blockage scores to the one or more candidate blocked regions based on region-associations among the one or more candidate blocked regions' in the plurality of images, and
in response to a determination that one of the blockage scores is above a predetermined blockage threshold, transmitting a blockage alarm signal to the alarm.

### Claim 17 (depends on 16)

The system as recited in the claim 16, wherein the identifying of the one or more candidate blocked regions comprises:
for an image selected from the plurality of images, dividing the image into a corresponding plurality of processing units (PUs);
identifying a plurality of blocked PUs from the plurality of PUs based on a subset of prior images in the plurality of images that are captured prior to the image; and
generating the one or more candidate blocked regions by clustering the plurality of blocked PUs.

### Claim 18 (depends on 17)

The system as recited in the claim 17, wherein the identifying of the plurality of blocked PUs comprises:
generating a feature value image based on feature values extracted from the plurality of PUs;
generating a feature value threshold based on the feature value image;
for a PU in the plurality of PUs, in response to a determination that the PU's corresponding feature value is above the feature value threshold, identifying the PU as an unblocked PU; and
identifying the plurality of blocked PUs based on unblocked PUs in the plurality of PUs.

### Claim 19 (depends on 17)

The system as recited in the claim 17, wherein the identifying of the plurality of blocked PUs comprises:
generating a codebook based on the set of prior images, wherein the codebook includes a codebook box containing minimum feature values extracted from the set of prior images; and
for a specific PU selected from the plurality of PUs, in response to a determination that the specific PU's feature value is within the codebook box's range, identifying the specific PU as one of the blocked PUs.

### Claim 20 (depends on 17)

The system as recited in the claim 17, wherein the identifying of the plurality of blocked PUs comprises:
generating a road-surface waveform based on feature values associated with road surfaces;
for a specific PU selected from the plurality of PUs, in response to a determination that the specific PU's feature value is within a predetermined range of the road-surface waveform, identifying the specific PU as an unblocked PU; and
identifying the plurality of blocked PUs based on unblocked PUs in the plurality of PUs.