# US 2021/0342628 A1 · Bale detection and classification using stereo cameras

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342628 A1
- Document ID: US20210342628A1
- Status: Published application
- Kind: A1
- Application number: 17/238,792
- Filed: 2021-04-23
- Published: 2021-11-04
- Earliest claimed: 2016-05-19
- Assignees: Vermeer Manufacturing Company
- Applicants: Vermeer Manufacturing Company
- Inventors: Jake Fevold; Edmond Dupont; Kristopher Kozak; Darin L. Dux; Curt Graham; Kent Thompson
- Web page: https://patentgrep.com/patent/US20210342628A1

## CPC

- A01D 90/08: with bale-forming means additionally used for loading; with means for picking-up bales and transporting them into the vehicle
- G06T 7/593: from stereo images
- G06V 10/446: {using Haar-like filters, e.g. using integral image techniques}
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06T 2207/10012: Stereo images
- G06T 2207/30188: Vegetation; Agriculture
- A01D: HARVESTING; MOWING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus comprises a sensor comprising a left camera and a right camera. A processor is coupled to the sensor. The processor is configured to produce an image and disparity data for the image, and search for a vertical object within the image using the disparity data. The processor is also configured to determine whether the vertical object is a bale of material using the image, and compute an orientation of the bale relative to the sensor using the disparity data. The sensor and processor can be mounted for use on an autonomous bale mover comprising an integral power system, a ground-drive system, a bale loading system, and a bale carrying system.

## Claims

### Claim 1

A method, comprising
scanning a region of land using a sensor, the scanned region of land comprising ground and one or more objects, the sensor mounted to an autonomous bale mover including an integral power system, a ground-drive system, a bale loading system, and a bale carrying system;
producing, using a processor coupled to the sensor and mounted to the autonomous bale mover, a three-dimensional (3-D) point cloud;
searching for a vertical object within the 3-D point cloud;
determining that the vertical object within the 3-D point cloud is a bale of material; and
computing an orientation of the bale relative to the sensor.

### Claim 2 (depends on 1)

The method of claim 1, wherein the sensor comprises at stereo cameras, a laser scanner, and/or a LIDAR scanner.

### Claim 3 (depends on 1)

The method of claim 1, wherein searching for the vertical object comprises searching for a vertical object having a height relative to the ground that is greater than a threshold.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining that the vertical object is the bale comprises:
extracting features of the vertical object;
classifying the vertical object using the extracted features and a classifier; and
determining that the vertical object is the bale in response to the classifier successfully classifying the vertical object as the bale.

### Claim 5 (depends on 1)

The method of claim 1, wherein computing bale orientation comprises computing position of the bale relative to the sensor using 3-D point cloud data.

### Claim 6 (depends on 1)

The method of claim 1, wherein computing bale orientation comprises:
computing three-dimensional points (X, Y, Z) for the bale within the image using 3-D point cloud data;
projecting X and Z coordinates of the three-dimensional points to a two-dimensional (X-Z) plane corresponding to a top-down view of the bale;
determining a face of the bale and a side of the bale using the two-dimensional plane; and computing the orientation of the bale relative to the sensor using the face of the bale.

### Claim 7 (depends on 6)

The method of claim 6, wherein determining the face and side of the bale comprises:
generating a first best fit line through points in the X-plane;
generating a second best fit line through points in the Z-plane; and
determining which of the first and second best fit lines represents the face of the bale.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
storing orientation and a position of the bale by a world model; and
updating the orientation and position of the bale in the world model in response to subsequent scanning of the bale by the sensor.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
receiving current orientation and current position of the bale by a world model; determining variability of the current orientation and current position relative to orientation and position data previously stored in the world model for the bale; and
updating the orientation and position of the bale in the world model to include the current orientation and current position if the variability does not exceed a threshold.

### Claim 10

An apparatus, comprising:
a sensor configured to scan a region of land comprising ground and one or more objects, the sensor mounted to an autonomous bale mover including an integral power system, a ground-drive system, a bale loading system, and a bale carrying system; and
a processor coupled to the sensor and mounted to the autonomous bale mover and configured to:
produce a three-dimensional (3-D) point cloud;
search for a vertical object within the 3-D point cloud;
determine that the vertical object within the 3-D point cloud is a bale of material; and
compute an orientation of the bale relative to the sensor.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the sensor comprises stereo cameras.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein the sensor comprises a laser scanner.

### Claim 13 (depends on 10)

The apparatus of claim 10, wherein the sensor comprises a LIDAR scanner.

### Claim 14 (depends on 10)

The apparatus of claim 10, wherein the processor is configured to search for a vertical object having a height relative to the ground that is greater than a threshold.

### Claim 15 (depends on 10)

The apparatus of claim 10, wherein the processor is configured to:
extract features of the vertical object;
classify the vertical object using the extracted features and a classifier; and
determine that the vertical object is the bale in response to the classifier successfully classifying the vertical object as the bale.

### Claim 16 (depends on 10)

The apparatus of claim 10, wherein the processor is configured to compute position of the bale relative to the sensor using 3-D point cloud data.

### Claim 17 (depends on 10)

The apparatus of claim 10, wherein the processor is configured to:
compute three-dimensional points (X, Y, Z) for the bale within the image using 3-D point cloud data;
project X and Z coordinates of the three-dimensional points to a two-dimensional (X-Z) plane corresponding to a top-down view of the bale;
determine a face of the bale and a side of the bale using the two-dimensional plane;
and compute the orientation of the bale relative to the sensor using the face of the bale.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein the processor is configured to:
generate a first best fit line through points in the X-plane;
generate a second best fit line through points in the Z-plane; and
determine which of the first and second best fit lines represents the face of the bale.

### Claim 19 (depends on 10)

The apparatus of claim 10, wherein the processor is configured to:
store orientation and a position of the bale by a world model; and
update the orientation and position of the bale in the world model in response to subsequent scanning of the bale by the sensor.

### Claim 20 (depends on 10)

The apparatus of claim 10, wherein the processor is configured to:
receive current orientation and current position of the bale by a world model;
determine variability of the current orientation and current position relative to orientation and position data previously stored in the world model for the bale; and
update the orientation and position of the bale in the world model to include the current orientation and current position if the variability does not exceed a threshold.
