# US 2023/0068848 A1 · Systems and methods for vehicle camera obstruction detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0068848 A1
- Document ID: US20230068848A1
- Status: Published application
- Kind: A1
- Application number: 17/411,403
- Filed: 2021-08-25
- Published: 2023-03-02
- Earliest claimed: 2021-08-25
- Assignees: Argo AI, LLC
- Applicants: Argo AI, LLC
- Inventors: Hatem Alismail
- Web page: https://patentgrep.com/patent/US20230068848A1

## CPC

- B60S 1/566: {including wiping devices (B60S1/563, B60S1/583 take precedence)}
- G06T 7/11: Region-based segmentation
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- B60S: SERVICING, CLEANING, REPAIRING, SUPPORTING, LIFTING, OR MANOEUVRING OF VEHICLES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR

## Abstract

A system such as an autonomous vehicle’s perception system will identify and classify an obstruction in a field of view of an image capturing device. The system will receive a sequence of image frames from the image capturing device. For each of the image frames, the system will segment the image frame into a regions of interest (ROIs), and the system will use a classifier to assign a classification to each ROI. The classification indicates whether the ROI is clear or obstructed. The system will aggregate the classifications for each ROI to determine an aggregate classification. When an obstructed classification persists for a threshold number of image frames, the system will classify the image capturing device as obstructed, and it will generate a function request that, when executed, will cause a system of which the image capturing device is a component to perform a function.

## Claims

### Claim 1

A method of identifying and classifying an obstruction in a field of view of an image capturing device, the method comprising, by a processor:
receiving a first sequence of image frames, wherein each image frame represents a field of view of an image capturing device;
for each of the image frames in the first sequence:
segmenting the image frame into a plurality of regions of interest (ROIs), and
using a classifier to assign a classification to each ROI, wherein the classification indicates whether the ROI is clear or obstructed;
aggregating the classifications for each ROI to determine an aggregate classification for each ROI; and
in response to an obstructed classification persisting for a threshold number of image frames:
classifying the image capturing device as obstructed, and
generating a function request that, when executed, will cause a system of which the image capturing device is a component to perform a function.

### Claim 2 (depends on 1)

The method of claim 1, wherein the function request includes one or more of the following:
a command to trigger a cleaning cycle that will cause a cleaning system to direct air or water toward the image capturing device to clean the image capturing device; or
a command to trigger a cleaning cycle that will cause a cleaning system to wipe image capturing device to clean the image capturing device.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the image capturing device is a component of an autonomous vehicle; and
the function request comprises a command to perform one or more of the following:
cause a motion planning system of the autonomous vehicle to move to a parking location, or
alert a human operator to take over operation of the autonomous vehicle.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
generating a confidence score for the aggregate classification; and
classifying the image capturing device as obstructed and generating the function request is also responsive to the confidence score for the aggregate classification exceeding a threshold.

### Claim 5 (depends on 4)

The method of claim 4, wherein generating the confidence score comprises generating a score that is a function of one or more of the following:
a number of the image frames in the sequence in which the obstruction was detected;
a number of spatially adjacent ROIs that were classified as obstructed; or
a location within the image frames at which obstructed ROIs were positioned.

### Claim 6 (depends on 1)

The method of claim 1, wherein classifying the image capturing device as obstructed also comprises classifying an obstruction in the image frames as either a translucent obstruction or an opaque obstruction.

### Claim 7 (depends on 1)

The method of claim 1, further comprising, for each of the image frames in the first sequence:
using the classifier to assign an obstruction type to each ROI which the classifier identifies as obstructed; and
when generating the function request, selecting a command that corresponds to the obstruction type, and including the selected command in the function request.

### Claim 8 (depends on 1)

The method of claim 1, further comprising, when the region of interest for the series of image frames is not classified as an obstruction:
continuing to receive additional image frames, segment each additional image frame into additional ROIs, use the classifier to assign an additional classification to each additional ROI, and aggregate the additional classifications until an obstructed classification persists for the threshold number of image frames.

### Claim 9

A system, comprising:
one or more image capturing devices; and
a computing device that includes a processor and memory, wherein the memory includes instructions that, when executed by the processor, will cause the processor to:
receive a first sequence of image frames, wherein each image frame represents a field of view of an image capturing device,
for each of the image frames in the first sequence:
segment the image frame into a plurality of regions of interest (ROIs), and
use a classifier to assign a classification to each ROI, wherein the classification indicates whether the ROI is clear or obstructed,
aggregate the classifications for each ROI to determine an aggregate classification for each ROI, and
in response to an obstructed classification persisting for a threshold number of image frames:
classify the image capturing device as obstructed, and
generate a function request that, when executed, will cause a system of which the image capturing device is a component to perform a function.

### Claim 10 (depends on 9)

The system of claim 9, wherein the instructions to generate the function request comprise instructions to generate one or more of the following:
a command to trigger a cleaning cycle that will cause a cleaning system to direct air or water toward the image capturing device to clean the image capturing device; or
a command to trigger a cleaning cycle that will cause a cleaning system to wipe image capturing device to clean the image capturing device.

### Claim 11 (depends on 9)

The system of claim 9, wherein:
the system is integral with an autonomoue vehicle;
the image capturing device is a component of the autonomous vehicle; and
the instructions to generate the function request comprise instructions to generate a command to perform one or more of the following:
cause a motion planning system of the autonomous vehicle to move to a parking location, or
alert a human operator to take over operation of the autonomous vehicle.

### Claim 12 (depends on 9)

The system of claim 9:
further comprising additional instructions that, when executed by the processor, will cause the processor to generate a confidence score for the aggregate classification, and
wherein the instructions to classify the image capturing device as obstructed are also configured to do so responsive to the confidence score for the aggregate classification exceeding a threshold.

### Claim 13 (depends on 12)

The system of claim 12, wherein the instructions to generate the confidence score comprise instructions to generate a score that is a function of one or more of the following:
a number of the image frames in the sequence in which the obstruction was detected;
a number of spatially adjacent ROIs that were classified as obstructed; or
a location within the image frames at which obstructed ROIs were positioned.

### Claim 14 (depends on 9)

The system of claim 9, wherein the instructions to classify the image capturing device as obstructed also comprise instructions to classify an obstruction in the image frames as either a translucent obstruction or an opaque obstruction.

### Claim 15 (depends on 9)

The system of claim 9, further comprising additional instructions to, for each of the image frames in the first sequence:
use the classifier to assign an obstruction type to each ROI which the classifier identifies as obstructed; and
when generating the function request, select a command that corresponds to the obstruction type, and include the selected command in the function request.

### Claim 16 (depends on 1)

The system of claim 1, further comprising additional instructions to, in response to the region of interest for the series of image frames not being classified as an obstruction:
receive additional image frames;
segment each additional image frame into additional ROIs;
use the classifier to assign an additional classification to each additional ROI; and
aggregate the additional classifications until an obstructed classification persists for the threshold number of image frames.

### Claim 17

A computer program product comprising a computer-readable medium that includes programming instructions that, when executed by a processor, will cause the processor to:
receive a first sequence of image frames, wherein each image frame represents a field of view of an image capturing device;
for each of the image frames in the first sequence:
segment the image frame into a plurality of regions of interest (ROIs), and
use a classifier to assign a classification to each ROI, wherein the classification indicates whether the ROI is clear or obstructed,
aggregate the classifications for each ROI to determine an aggregate classification for each ROI; and
in response to an obstructed classification persisting for a threshold number of image frames:
classify the image capturing device as obstructed, and
generate a function request that, when executed, will cause a system of which the image capturing device is a component to perform a function.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein the instructions to generate the function request comprise instructions to generate one or more of the following:
a command to trigger a cleaning cycle that will cause a cleaning system to direct air or water toward the image capturing device to clean the image capturing device; or
a command to trigger a cleaning cycle that will cause a cleaning system to wipe image capturing device to clean the image capturing device.

### Claim 19 (depends on 17)

The computer program product of claim 17:
further comprising additional instructions that, when executed by the processor, will cause the processor to generate a confidence score for the aggregate classification, and
wherein the instructions to classify the image capturing device as obstructed are also configured to do so responsive to the confidence score for the aggregate classification exceeding a threshold.

### Claim 20 (depends on 9)

The system of claim 9, wherein the instructions to classify the image capturing device as obstructed also comprise instructions to classify an obstruction in the image frames as either a translucent obstruction or an opaque obstruction.
