# US 2024/0004087 A1 · Center Path Estimator for Agricultural Windrows

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004087 A1
- Document ID: US20240004087A1
- Status: Published application
- Kind: A1
- Application number: 17/856,326
- Filed: 2022-07-01
- Published: 2024-01-04
- Earliest claimed: 2022-07-01
- Assignees: Southwest Research Institute
- Applicants: Southwest Research Institute
- Inventors: Edmund M DuPont; Prasanna R. Kolar
- Web page: https://patentgrep.com/patent/US20240004087A1

## CPC

- G01B 11/24: for measuring contours or curvatures
- G01S 19/45: by combining measurements of signals from the satellite radio beacon positioning system with a supplementary measurement
- G01B 11/22: for measuring depth
- G01C 21/20: Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence)
- G06N 3/09: Supervised learning
- G06T 7/11: Region-based segmentation
- G06T 7/593: from stereo images
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/188: {Vegetation}
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- A01B 69/008: {automatic}
- A01F 15/07: Rotobalers, i.e. machines for forming cylindrical bales by winding and pressing
- G06T 2207/10012: Stereo images
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10152: Varying illumination
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 23/667: Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A01B: SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL (making or covering furrows or holes for sowing, planting or manuring A01C5/00; mowers convertible to soil working apparatus or capable of soil working A01D42/04; mowers combined with soil working implements A01D43/12; soil working for engineering purposes E01, E02, E21)
- A01F: PROCESSING OF HARVESTED PRODUCE; HAY OR STRAW PRESSES; DEVICES FOR STORING AGRICULTURAL OR HORTICULTURAL PRODUCE (devices for topping or skinning onions or flower bulbs A23N15/08)

## Abstract

A windrow path detector for use by baling equipment. A camera system is mounted on the baling equipment and provides image data representing a stereo view of a windrow ahead of the baling equipment. A pre-processor to generates a depth image. A neural network receives the depth image and the image data from at least one camera, and is trained to use that input to generate a predicted center line over the windrow. A center path process uses the predicted center line to define a navigation path, which it delivers to a navigation system on-board the baling equipment.

## Claims

### Claim 1

A windrow path detector for use by baling equipment, the baling equipment being equipped with a navigation system, comprising:
a camera system mounted on the baling equipment and having at least two spaced cameras, the cameras operable to provide image data representing a stereo view of a windrow ahead of the baling equipment;
an image data pre-processor operable to receive the image data from each camera and to generate a depth image;
a neural network operable to receive input data comprising the depth image and the image data from at least one camera, the neural network being trained to use the input data to generate a predicted center line over the windrow; and
a center path process operable to use the predicted center line to define a navigation path, using a spline fit and converting pixels into three-dimensional points relative to the cameras and projected onto the ground surface;
the center path process further operable to deliver the navigation path to the navigation system.

### Claim 2 (depends on 1)

The windrow path detector of claim 1, wherein the camera system comprises back-illuminated pixel cameras.

### Claim 3 (depends on 1)

The windrow path detector of claim 1, wherein the camera system is operable in daylight and low-light conditions.

### Claim 4 (depends on 1)

The windrow path detector of claim 1, wherein the windrow is silage material.

### Claim 5 (depends on 1)

The windrow path detector of claim 1, wherein the windrow is compost material.

### Claim 6 (depends on 1)

The windrow path detector of claim 1, wherein the navigation system is a GPS (global positioning system) navigation system.

### Claim 7

A method of training a neural network to perform windrow path detector for use by baling equipment, comprising:
driving a vehicle having a camera system over a number of test windrows, the camera system having at least two spaced cameras operable to provide image data representing a stereo view of each windrow when ahead of the vehicle;
combining the image data with depth data to provide a perspective image of the windrow
using a navigation system of the vehicle to create localization data representing a driven path over the windrows;
overlaying the perspective image data with the localization data, thereby generating labeled image data;
inputting the labeled image data to the neural network; and
performing a training process on the neural network to estimate the center of the windrow.

### Claim 8 (depends on 7)

The method of claim 7, wherein the driving step is performed during daytime and low-light conditions.

### Claim 9 (depends on 7)

The method of claim 7, wherein the camera system comprises back-illuminated pixel cameras.

### Claim 10 (depends on 7)

The method of claim 7, wherein the navigation system is a GPS (global positioning system) navigation system.
