# US 2019/0379847 A1 · System and method for monitoring sensor performance on an agricultural machine

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0379847 A1
- Document ID: US20190379847A1
- Status: Published application
- Kind: A1
- Application number: 16/001,405
- Filed: 2018-06-06
- Published: 2019-12-12
- Earliest claimed: 2018-06-06
- Assignees: CNH Industrial Canada, Ltd.; Autonomous Solutions, Inc.
- Applicants: CNH Industrial Canada, Ltd.; Autonomous Solutions, Inc.
- Inventors: Luca Ferrari; John H. Posselius; James W. Henry; Taylor C. Bybee; Bret T. Turpin
- Web page: https://patentgrep.com/patent/US20190379847A1

## CPC

- H04N 25/683: by defect estimation performed on the scene signal, e.g. real time or on the fly detection
- A01B 76/00: Parts, details or accessories of agricultural machines or implements, not provided for in groups A01B51/00 - A01B75/00
- B60S 1/603: {the operation of at least a part of the cleaning means being controlled by electric means}
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/90: Determination of colour characteristics
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
- B60S 1/04: Wipers or the like, e.g. scrapers
- B60S 1/566: {including wiping devices (B60S1/563, B60S1/583 take precedence)}
- G06T 2207/30168: Image quality inspection
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A01B: SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL (making or covering furrows or holes for sowing, planting or manuring A01C5/00; mowers convertible to soil working apparatus or capable of soil working A01D42/04; mowers combined with soil working implements A01D43/12; soil working for engineering purposes E01, E02, E21)
- B60S: SERVICING, CLEANING, REPAIRING, SUPPORTING, LIFTING, OR MANOEUVRING OF VEHICLES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

In one aspect, a system for monitoring sensor performance on an agricultural machine may include a controller configured to receive a plurality of images from the vision-based sensor mounted on an agricultural machine. The controller may be configured to determine an image parameter value associated with each of a plurality of pixels contained within each of the plurality of images. For each respective pixel of the plurality of pixels, the controller may be configured to determine a variance associated with the image parameter values for the respective pixel across the plurality of images. Furthermore, when the variance associated with the image parameter values for a given pixel of the plurality of pixels falls outside of a predetermined range, the controller may be configured to identify the given pixel as being at least one of obscured or inoperative.

## Claims

### Claim 1

A system for monitoring sensor performance on an agricultural machine, the system comprising:
an agricultural machine;
a vision-based sensor mounted on the agricultural machine, the vision-based sensor being configured to capture a plurality of images; and
a controller communicatively coupled to the vision-based sensor, the controller being configured to:
receive the plurality of images from the vision-based sensor;
determine an image parameter value associated with each of a plurality of pixels contained within each of the plurality of images;
for each respective pixel of the plurality of pixels, determine a variance associated with the image parameter values for the respective pixel across the plurality of images; and
when the variance associated with the image parameter values for a given pixel of the plurality of pixels falls outside of a predetermined range, identify the given pixel as being at least one of obscured or inoperative.

### Claim 2 (depends on 1)

The system of claim 1, wherein the image parameter value corresponds to at least one of an intensity or a color of the light associated with each of the plurality of pixels.

### Claim 3 (depends on 1)

The system of claim 1, wherein the controller is configured to determine the variance in the image parameter values by determining a differential between the image parameter values or determining a standard deviation of the image parameter values.

### Claim 4 (depends on 3)

The system of claim 3, wherein, when determining the variance in the image parameter values, the controller is further configured to weight each image parameter value based on a time at which the corresponding image was captured.

### Claim 5 (depends on 1)

The system of claim 1, wherein the controller is configured to identify the given pixel as being obscured or inoperative when an intensity of the light detected by the given pixel falls below a predetermined intensity threshold.

### Claim 6 (depends on 1)

The system of claim 1, wherein the controller is further configured to identify the vision-based sensor as being obscured or inoperative when a given number of individual pixels of the plurality of pixels that have been identified as being obscured or inoperative exceeds a predetermined threshold amount.

### Claim 7 (depends on 1)

The system of claim 1, wherein the controller is further configured to:
aggregate the individual pixels of the plurality of pixels that have been identified as being obscured or inoperative into one or more pixel groups; and
identify the vision-based sensor as being obscured or inoperative when a density of the individual pixels within the one or more the pixel groups exceeds a predetermined threshold density value.

### Claim 8 (depends on 6)

The system of claim 6, wherein the controller is further configured to initiate a control action when it is determined that the vision-based sensor is obscured or inoperative.

### Claim 9 (depends on 8)

The system of claim 8, wherein the control action is associated with notifying an operator of the agricultural machine that the vision-based sensor is obscured or inoperative.

### Claim 10 (depends on 8)

The system of claim 8, wherein the control action is associated with reducing a ground speed of the agricultural machine.

### Claim 11 (depends on 8)

The system of claim 8, wherein the control action is associated with automatically cleaning the vision-based sensor.

### Claim 12 (depends on 1)

The system of claim 1, wherein the vision-based sensor comprises at least one of a camera or a Light Detection and Ranging (LIDAR) sensor.

### Claim 13

A method for monitoring sensor performance on an agricultural machine, the method comprising:
receiving, with a computing device, the plurality of images from the vision-based sensor;
determining, with the computing device, an image parameter value associated with each of a plurality of pixels contained within each of the plurality of images;
for each respective pixel of the plurality of pixels, determining, with the computing device, a variance associated with the image parameter values for the respective pixel across the plurality of images; and
when the variance associated with the image parameter values for a given pixel of the plurality of pixels falls outside of a predetermined range, identifying, with the computing device, the given pixel as being at least one of obscured or inoperative;
determining, with the computing device, when the vision-based sensor is obscured or inoperative based on a number or a density of individual pixels of the plurality of pixels that have been identified as being obscured or inoperative; and
initiating, with the computing device, a control action when it is determined that the vision-based sensor is obscured or inoperative.

### Claim 14 (depends on 13)

The method of claim 13, wherein the image parameter value is corresponds to at least one of an intensity or a color of the light associated with each of the plurality of pixels.

### Claim 15 (depends on 13)

The method of claim 13, wherein determining the variance in the image parameter values comprises determining, with the computing device, a differential between the image parameter values or determining, with the computing device, a standard deviation of the image parameter values.

### Claim 16 (depends on 15)

The method of claim 15, wherein determining the variance in the image parameter values comprises weighting, with the computing device, each image parameter value based on a time at which the corresponding image was captured.

### Claim 17 (depends on 13)

The method of claim 13, further comprising:
identifying, with the computing device, the given pixel as being obscured or inoperative when an intensity of the light detected by the given pixel falls below a predetermined intensity threshold.

### Claim 18 (depends on 13)

The method of claim 13, wherein determining when the vision-based sensor is obscured or inoperative comprises identifying, with the computing device, the vision-based sensor as being obscured or inoperative comprises when a given number of individual pixels of the plurality of pixels that have been identified as being obscured or inoperative exceeds a predetermined threshold amount.

### Claim 19 (depends on 13)

The method of claim 13, wherein determining when the vision-based sensor is obscured or inoperative comprises:
aggregating, with the computing device, the individual pixels of the plurality of pixels that have been identified as being obscured or inoperative into one or more pixel groups; and
identifying, with the computing device, the vision-based sensor as being obscured or inoperative when a density of the individual pixels within the one or more the pixel groups exceeds a predetermined threshold density value.

### Claim 20 (depends on 8)

The system of claim 8, wherein the control action is associated with reducing a ground speed of the agricultural machine.
