# US 2019/0294914 A1 · Bale detection and classification using stereo cameras

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0294914 A1
- Document ID: US20190294914A1
- Status: Published application
- Kind: A1
- Application number: 16/302,427
- Filed: 2017-05-16
- Published: 2019-09-26
- Earliest claimed: 2016-05-19
- Assignees: Vermeer Manufacturing Company
- Applicants: Vermeer Manufacturing Company
- Inventors: Jake Fevold; Edmond Dupont; Kristopher Kozak; Darin L. Dux; Curt Graham; Kent Thompson
- Web page: https://patentgrep.com/patent/US20190294914A1

## CPC

- A01D 90/08: with bale-forming means additionally used for loading; with means for picking-up bales and transporting them into the vehicle
- G06T 7/593: from stereo images
- G06V 10/446: {using Haar-like filters, e.g. using integral image techniques}
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06T 2207/10012: Stereo images
- G06T 2207/30188: Vegetation; Agriculture
- A01D: HARVESTING; MOWING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus comprises a sensor (102) comprising a left camera (102a) and a right camera (102b). A processor (104) is coupled to the sensor. The processor is configured to produce an image and disparity data for the image, and search for a vertical object (122a, 122b) within the image using the disparity data. The processor is also configured to determine whether the vertical object is a bale of material using the image, and compute an orientation of the bale relative to the sensor using the disparity data. The sensor and processor can be mounted for use on an autonomous bale mover comprising an integral power system, a ground-drive system, a bale loading system, and a bale carrying system.

## Claims

### Claim at position 1

1-95. (canceled)

### Claim 96

A method, comprising:
scanning a region of land using a sensor comprising stereo cameras;
producing, by the sensor, an image and disparity data for the image;
searching for a vertical object within the image using the disparity data;
determining that the vertical object is a bale of material using the image; and
computing an orientation of the bale relative to the sensor using the disparity data.

### Claim 97 (depends on 96)

The method of claim 96, further comprising:
producing modified disparity data by removing disparity data corresponding to the ground in the image; and
searching for the vertical object within the region using the modified disparity data.

### Claim 98 (depends on 96)

The method of claim 96, wherein:
searching for the vertical object comprises scanning the image using a detection window having a predetermined size in terms of pixels; and
the predetermined size of the detection window corresponds to a size of the bale at a given distance separating the vertical object from the sensor.

### Claim 99 (depends on 96)

The method of claim 96, wherein determining that the vertical object is the bale comprises:
extracting features of the vertical object;
classifying the vertical object using the extracted features and a plurality of classifiers; and
determining that the vertical object is the bale in response to each of the plurality of classifiers successfully classifying the vertical object as the bale.

### Claim 100 (depends on 96)

The method of claim 96, wherein determining that the vertical object is the bale comprises:
classifying, by a first classifier, the vertical object using first features of the object; and
if the first classifier indicates the vertical object is likely the bale, classifying, by a second classifier, the vertical object using second features of the object.

### Claim 101 (depends on 100)

The method of claim 100, wherein:
the first classifier comprises a first support vector machine;
the first features are Haar features;
the second classifier comprises a second support vector machine; and
the second features are HOG (Histogram of Oriented Gradients) features.

### Claim 102 (depends on 96)

The method of claim 96, wherein computing bale orientation comprises computing position of the bale relative to the sensor using the disparity data.

### Claim 103 (depends on 96)

The method of claim 96, wherein computing bale orientation comprises:
computing three-dimensional points (X, Y, Z) for the bale within the image using the disparity data;
projecting X and Z coordinates of the three-dimensional points to a two-dimensional (X-Z) plane corresponding to a top-down view of the bale;
determining a face of the bale and a side of the bale using the two-dimensional plane; and
computing the orientation of the bale relative to the sensor using the face of the bale.

### Claim 104 (depends on 103)

The method of claim 103, wherein determining the face and side of the bale comprises:
generating a first best fit line through points in the X-plane;
generating a second best fit line through points in the Z-plane; and
determining which of the first and second best fit lines represents the face of the bale.

### Claim 105 (depends on 104)

The method of claim 104, wherein:
determining which of the first and second best fit lines represents the face of the bale comprises determining disparity data variation for the first and second best fit lines; and
the best fit line with the smallest variation corresponds to the face of the bale.

### Claim 106 (depends on 96)

The method of claim 96, further comprising:
storing orientation and a position of the bale by a world model; and
updating the orientation and position of the bale in the world model in response to subsequent imaging of the bale by the sensor.

### Claim 107 (depends on 96)

The method of claim 96, further comprising:
receiving current orientation and current position of the bale by a world model;
determining variability of the current orientation and current position relative to orientation and position data previously stored in the world model for the bale; and
updating the orientation and position of the bale in the world model to include the current orientation and current position if the variability does not exceed a threshold.

### Claim 108

An apparatus, comprising:
a sensor comprising a left camera and a right camera; and
a processor coupled to the sensor and configured to:
produce an image and disparity data for the image;
search for a vertical object within the image using the disparity data;
determine that the vertical object is a bale of material using the image; and
compute an orientation of the bale relative to the sensor using the disparity data.

### Claim 109 (depends on 108)

The apparatus of claim 108, wherein the processor is configured to:
produce modified disparity data by removing disparity data corresponding to the ground in the image; and
search for the vertical object within the region using the modified disparity data.

### Claim 110 (depends on 108)

The apparatus of claim 108, wherein:
the processor is configured to search for the vertical object by scanning the image using a detection window having a predetermined size in terms of pixels; and
the predetermined size of the detection window corresponds to a size of the bale at a given distance separating the vertical object from the sensor.

### Claim 111 (depends on 108)

The apparatus of claim 108, wherein the processor is configured to determine that the vertical object is the bale by:
extracting features of the vertical object;
classifying the vertical object using the extracted features and a plurality of classifiers; and
determining that the vertical object is the bale in response to each of the plurality of classifiers successfully classifying the vertical object as the bale.

### Claim 112 (depends on 108)

The apparatus of claim 108, wherein the processor is configured to determine that the vertical object is the bale by:
classifying, by a first classifier, the vertical object using first features of the object; and
if the first classifier indicates the vertical object is likely the bale, classifying, by a second classifier, the vertical object using second features of the object.

### Claim 113 (depends on 112)

The apparatus of claim 112, wherein:
the first classifier comprises a first support vector machine;
the first features are Haar features;
the second classifier comprises a second support vector machine; and
the second features are HOG (Histogram of Oriented Gradients) features.

### Claim 114 (depends on 108)

The apparatus of claim 108, wherein the processor is configured to compute bale orientation by computing position of the bale relative to the sensor using the disparity data.

### Claim 115 (depends on 108)

The apparatus of claim 108, wherein the processor is configured to compute bale orientation by:
computing three-dimensional points (X, Y, Z) for the bale within the image using the disparity data;
projecting X and Z coordinates of the three-dimensional points to a two-dimensional (X-Z) plane corresponding to a top-down view of the bale;
determining a face of the bale and a side of the bale using the two-dimensional plane; and
computing the orientation of the bale relative to the sensor using the face of the bale.

### Claim 116 (depends on 115)

The apparatus of claim 115, wherein the processor is configured to determine the face and side of the bale by:
generating a first best fit line through points in the X-plane;
generating a second best fit line through points in the Z-plane; and
determining which of the first and second best fit lines represents the face of the bale.

### Claim 117 (depends on 116)

The apparatus of claim 116, wherein the processor is configured to:
determine which of the first and second best fit lines represents the face of the bale by determining disparity data variation for the first and second best fit lines; and
the best fit line with the smallest variation corresponds to the face of the bale.

### Claim 118 (depends on 108)

The apparatus of claim 108, wherein the processor is configured to:
store orientation and a position of the bale by a world model; and
update the orientation and position of the bale in the world model in response to subsequent imaging of the bale by the sensor.

### Claim 119 (depends on 108)

The apparatus of claim 108, wherein the processor is configured to:
receive current orientation and current position of the bale by a world model;
determine variability of the current orientation and current position relative to orientation and position data previously stored in the world model for the bale; and
update the orientation and position of the bale in the world model to include the current orientation and current position if the variability does not exceed a threshold.

### Claim 120 (depends on 108)

The apparatus of claim 108, further comprising an autonomous bale mover, wherein the sensor and processor are components of the bale mover.
