# US 7,610,123 B2 · Vision-aided system and method for guiding a vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 7,610,123 B2
- Document ID: US7610123B2
- Status: Granted
- Kind: B2
- Application number: 11/106,782
- Filed: 2005-04-15
- Published: 2006-07-06
- Granted: 2009-10-27
- Earliest claimed: 2005-01-04
- Assignees: Deere & Company
- Inventors: Shufeng Han; John Franklin Reid; Francisco Rovira-Más
- Web page: https://patentgrep.com/patent/US7610123B2

## CPC

- G01C 21/28: with correlation of data from several navigational instruments
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/0278: {using satellite positioning signals, e.g. GPS}
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

A method and system for guiding a vehicle comprises a location determining receiver for collecting location data for the vehicle. A vision module collects vision data for the vehicle. A location quality estimator estimates the location quality data for the location data during an evaluation time window. A vision module estimates vision quality data for the vision data during the evaluation time window. A supervisor module selects at least one of a location data weight and a vision data weight based on the quality data.

## Claims

### Claim 1

A method for guiding a vehicle, the method comprising:
collecting location data for the vehicle based on a location-determining receiver of a location module associated with the vehicle;
collecting vision data for the vehicle based on a vision module associated with the vehicle;
estimating a quality of the collected location data during an evaluation time window to form location quality data;
estimating a quality of the collected vision data during the evaluation time window to form vision quality data;
selecting at least one of a location data weight and a vision data weight to form a selection, wherein the location data weight is based on the location quality data and the vision data weight is based on the vision quality data; and
guiding the vehicle in accordance with the selection, wherein the selecting further comprises:
identifying a quality-mixing ratio based on respective input values associated with the vision quality data and the location quality data;
determining output values associated with the location data weight and the vision data weight based on the identified quality-mixing ratio; and
applying the vision data weight and the location data weight to determine relative contributions of the collected vision data and the collected location data to a vehicular guidance system for guiding the vehicle for a time interval.

### Claim 2 (depends on 1)

The method according to claim 1 wherein estimating a quality of the collected location data comprises detecting an off track error, a heading error, and a curvature associated with the location-determining receiver.

### Claim 3 (depends on 1)

The method according to claim 1 wherein estimating a quality of the collected vision data comprises detecting an off track error, a heading error, and a curvature associated with the vision module.

### Claim 4 (depends on 1)

The method according to claim 1 wherein the vision data weight and the location data weight are consistent with the following equation: αvision+αgps=1.

### Claim 5 (depends on 1)

The method according to claim 1 wherein:
identifying the quality-mixing ratio based on respective input values associated with the vision quality data and the location quality data, comprises identifying the quality-mixing ratio based on respective input values associated with the vision quality data, the location quality data, and curvature data; wherein
determining output values associated with the location data weight and the vision data weight based on the identified quality-mixing ratio, comprises determining output values associated with the location data weight, the vision data weight, and a curvature data weight based on the identified quality-mixing ratio; and wherein
applying the vision data weight and the location data weight to determine relative contributions of the collected vision data and the collected location data to a vehicular guidance system for guiding the vehicle for a time interval, comprises applying the vision data weight, the location data weight, and the curvature data weight to determine relative contributions of the collected vision data and the collected location data to the vehicular guidance system for guiding the vehicle for the time interval.

### Claim 6 (depends on 1)

The method according to claim 1 wherein the applying the vision data weight and the location data weight to determine relative contributions of the collected vision data and the collected location data to a vehicular guidance system for guiding the vehicle for a time interval, comprises:
generating an error control signal for steering the vehicle in accordance with the following equations: y=αvision×yvision+αgps×ygps, where y is the error control signal, αvision is the vision data weight, yvision is a vision error signal, αgps is the location data weight, and ygps is a location error signal.

### Claim 7 (depends on 6)

The method according to claim 6 wherein the y, yvision, ygps, αgps, and αvision are multidimensional vectors in accordance with the following expressions:
y
=
[
E
off
E
head
]
,
Eoff is an aggregate off-track error from an aggregation of error data from the vision module and the location module, Ehead is an aggregate heading error from the aggregation of the error data from the vision module and the location module;
a
vision
=
[
α
off
⁢
⁢
vision
α
head_vision
]
,
where αvision is a vision data weight matrix, αoff—vision is the vision data weight for off track error data, and αhead—vision is the vision data weight for heading error data;
a
gps
=
[
α
off_gps
α
head_gps
]
,
where αgps is a location data weight matrix, αoff—gps is the location data weight for off track error data, and αhead—gps is the location data weight for heading error data;
y
gps
=
[
E
off_gps
E
head_gps
]
,
where Eoff—gps is an off-track error estimated by the location module, and Ehead—gps is a heading error estimated by the location module; and
y
vision
=
[
E
off_vision
E
head_vision
]
,
where Eoff—vision is an off track error estimated by the vision module and Ehead—vision is a heading error estimated by the vision module.

### Claim 8 (depends on 5)

The method according to claim 5 wherein the applying the vision data weight, the location data weight and the curvature data weight to determine relative contributions of the collected vision data and the collected location data to the vehicular guidance system for guiding the vehicle for the time interval, comprises:
generating an error control signal for steering the vehicle in accordance with the following equation: y=αvision×yvision+αgps×ygps, where y is the error control signal, αvision is the vision data weight, yvision is a vision error signal, αgps is the location data weight, and ygps is a location error signal.

### Claim 9 (depends on 8)

The method according to claim 8 wherein the y, yvision, ygps, αvision, and αgps, are multidimensional vectors in accordance with the following expressions:
y
=
[
E
off
E
head
ρ
]
,
Eoff is the aggregate off-track error from the aggregation of error data from the vision module and the location module, Ehead is the aggregate heading error from an aggregation of the error data from the vision module and the location module and ρ is radius of curvature;
a
vision
=
[
α
off_vision
α
head_vision
α
curv_vision
]
,
where αvision is the aggregate vision data weight matrix, αoff—vision is the vision data weight for off-track error data, αhead—vision is the vision data weight for heading error data, and αcurv—vision is vision data weight for curvature error data;
y
vision
=
[
E
off_vision
E
head_vision
0
]
,
where Eoff—vision is the off track error estimated by the vision module and Ehead—vision is the heading error estimated by the vision module;
a
gps
=
[
α
off_gps
α
head_gps
α
curv_gps
]
,
where αgps is the aggregate location data weight matrix, αoff—gps is the location data weight for off-track error data, αhead—gps is the location data weight for heading error data, and αcurv—gps is the location data weight for curvature error data;
y
gps
=
[
E
off_gps
E
head_gps
ρ
gps
]
,
where Eoff—gps is the off-track error estimated by the location module, Ehead—gps is the heading error estimated by the location module, and ρgps is radius of curvature associated with the location module.

### Claim 10

A system for guiding a vehicle, the system comprising:
a location-determining receiver of location module associated with the vehicle for collecting location data for the vehicle;
a vision module associated with the vehicle for collecting vision data for the vehicle;
a location quality estimator for estimating a quality of the collected location data during an evaluation time window to form location quality data;
a vision quality estimator for estimating a quality of the collected vision data during the evaluation time window to form vision quality data; and
a selector for selecting at least one of a location data weight and a vision data weight to form a selection, wherein the location data weight is based on the location quality data and the vision data weight is based on the vision quality data; and
a vehicular guidance system for guiding the vehicle in accordance with the selection, wherein the selector selects the at least one of the location data weight and the vision data weight to form the selection by:
identifying a quality-mixing ratio based on respective input values associated with the vision quality data and the location quality data;
determining output values associated with the location data weight, and the vision data weight based on the identified quality-mixing ratio; and
applying the vision data weight and the location data weight to determine relative contributions of the collected vision data and the collected location data to the vehicular guidance system for a time interval.

### Claim 11 (depends on 10)

The system according to claim 10 wherein the location quality estimator estimates the quality of the collected location data using an off-track error, a heading error, and a curvature associated with the location-determining receiver.

### Claim 12 (depends on 10)

The system according to claim 10 wherein the vision quality estimator estimates the quality of the collected vision data using an off-track error, a heading error, and a curvature associated with the vision module.

### Claim 13 (depends on 10)

The system according to claim 10 wherein the vision data weight and the location data weight are consistent with the following equation: αvision+αgps=1.

### Claim 14 (depends on 10)

The system according to claim 10 wherein the selector selects at least one of the location data weight and the vision data weight to form the selection by:
identifying the quality-mixing ratio based on respective input values associated with the vision quality data, the location quality data, and curvature data;
determining the output values associated with the location data weight, the vision data weight, and curvature data weight based on the identified quality-mixing ratio; and
applying the vision data weight, the location data weight, and the curvature data weight to determine relative contributions of the collected vision data and the collected location data to the vehicular guidance system for the time interval.

### Claim 15 (depends on 10)

The system according to claim 10 wherein the applying the vision data weight and the location data weight to determine relative contributions of the collected vision data and the collected location data to the vehicular guidance system for the time interval, comprises:
generating an error control signal for steering the vehicle in accordance with the following equation: y=αvision×yvision+αgps×ygps, where y is the error control signal, αvision is vision data weight, yvision is a vision error signal, αgps is the location data weight, and ygps is a location error signal.

### Claim 16 (depends on 15)

The system according to claim 15 wherein the y, yvision, ygps, αgps, and αvision are multidimensional vectors in accordance with the following expressions:
y
=
[
E
off
E
head
]
,
Eoff is the aggregate off-track error from the aggregation of error data from the vision module and the location module, Ehead is the aggregate heading error from the aggregation of the error data from the vision module and the location module;
α
vision
=
[
α
off_vision
α
head_vision
]
,
where αvision is the vision data weight matrix, αoff—vision is the vision data weight for off track error data, and αhead—vision is the vision data weight for heading error data;
α
gps
=
[
α
off
⁢
⁢
gps
α
head
⁢
⁢
gps
]
,
where αgps is the location data weight matrix, αoff—gps is the location data weight for off track error data, and αhead—gps is the location data weight for heading error data;
y
gps
=
[
E
off_gps
E
head
⁢
⁢
gps
]
,
where Eoff—gps is the off-track error estimated by the location module and Ehead—gps is the heading error estimated by the location module; and
y
vision
=
[
E
off_vision
E
head_vision
]
,
where Eoff—vision is the off track error estimated by the vision module and Ehead—vision is a heading error estimated by the vision module.

### Claim 17 (depends on 14)

The system according to claim 14 wherein the applying the vision data weight, the location data weight, and the curvature data weight to determine relative contributions of the collected vision data and the collected location data to the vehicular guidance system for the time interval, comprises:
generating an error control signal for steering the vehicle in accordance with the following equation: y=αvision×yvision+αgps×ygps, where y is the error control signal, αvision is the vision data weight, yvision is a vision error signal, αgps is the location data weight, and ygps is a location error signal.

### Claim 18 (depends on 17)

The system according to claim 17 wherein the y, yvision, ygps, αvision, and αgps are multidimensional vectors in accordance with the following expressions:
y
=
[
E
off
E
head
ρ
]
,
Eoff is the aggregate off-track error from the aggregation of error data from the vision module and the location module, Ehead is the aggregate heading error from an aggregation of the error data from the vision module and the location module and ρ is radius of curvature based on the vision module and the location module;
α
vision
=
[
α
off_vision
α
head_vision
α
curv_vision
]
,
where αvision is the aggregate vision data weight matrix, αoff—vision is the vision data weight for off-track error data, αhead—vision is the vision data weight for heading error data, and αcurv—vision is vision data weight for curvature error data;
y
vision
=
[
E
off_vision
E
head_vision
ρ
vision
]
,
where Eoff—vision, is the off track error estimated by the vision module and Ehead—vision is the heading error estimated by the vision module, and ρv is a radius of curvature associated with the vision module;
α
gps
=
[
α
off_gps
α
head_gps
α
curv_gps
]
,
where αgps is the aggregate location data weight matrix, αoff—gps is the location data weight for off-track error data, αhead—gps is the location data weight for heading error data, and αcurv—gps is the location data weight for curvature error data;
y
gps
=
[
E
off_gps
E
head_gps
ρ
gps
]
,
where Eoff—gps is the off-track error estimated by the location module, Ehead—gps is the heading error estimated by the location module, and ρgps is radius of curvature associated with the location module.