# US 2021/0333790 A1 · Using generated markings for vehicle control and object avoidance

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0333790 A1
- Document ID: US20210333790A1
- Status: Published application
- Kind: A1
- Application number: 16/859,033
- Filed: 2020-04-27
- Published: 2021-10-28
- Earliest claimed: 2020-04-27
- Assignees: Deere & Company
- Applicants: Deere & Company
- Inventors: Michael G. KEAN; Amy K. JONES; Brett S. GRAHAM; Nathaniel M. CZARNECKI; Gordon E. MILLER; Rachel BRUFLODT
- Web page: https://patentgrep.com/patent/US20210333790A1

## CPC

- G05D 1/0016: {characterised by the operator's input device (input arrangements for computing systems in general G06F3/00)}
- B60W 10/04: including control of propulsion units
- B60W 10/20: including control of steering systems
- B60W 60/0025: {specially adapted for specific operations}
- G01C 21/20: Instruments for performing navigational calculations (G01C21/24, G01C21/26 take precedence)
- G01C 21/3461: {Preferred or disfavoured areas, e.g. dangerous zones, toll or emission zones, intersections, manoeuvre types or segments such as motorways, toll roads or ferries}
- G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)}
- G05D 1/0234: {using optical markers or beacons (optical beacons per se G01S1/70)}
- G05D 1/0251: {extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593)}
- G05D 1/0253: {extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow (determining position or orientation from images G06T7/70)}
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)

## Abstract

A work machine has a backup camera that captures images of an area of a worksite behind the work machine. A controller identifies pre-defined markings in the worksite and localizes the pre-defined markings to the work machine, based on the images. A control signal generator generates control signals to automatically control the work machine based upon the localized markings.

## Claims

### Claim 1

A computer implemented method of controlling a mobile work machine, comprising:
detecting an attachment control operator input;
controlling operation of an attachment on the mobile work machine based on the detected attachment control operator input;
capturing an image of a worksite from a backup camera on the mobile work machine;
identifying an operator-generated marking, having a predefined characteristic, in the image;
localizing the predefined operator-generated marking to the mobile work machine; and
while controlling operation of the attachment on the mobile work machine based on the detected attachment control operator input, automatically controlling machine travel of the mobile work machine based on the localized, operator-generated marking.

### Claim 2 (depends on 1)

The computer implemented method of claim 1 wherein identifying an operator-generated marking comprises:
identifying a route line indicating a desired travel route of the mobile work machine.

### Claim 3 (depends on 2)

The computer implemented method of claim 2 wherein automatically controlling machine travel comprises:
detecting a travel command input commanding the mobile work machine to travel; and
controlling a propulsion subsystem and a steering subsystem on the mobile work machine so the mobile work machine travels along the desired travel route indicated by the route line.

### Claim 4 (depends on 3)

The computer implemented method of claim 3 wherein controlling the propulsion subsystem and the steering subsystem comprises:
detecting a target speed; and
controlling the propulsion subsystem and the steering subsystem on the mobile work machine so the mobile work machine travels along the desired travel route until a stop condition is detected.

### Claim 5 (depends on 4)

The computer implemented method of claim 4 wherein controlling the propulsion subsystem and the steering subsystem on the mobile work machine comprises:
detecting the stop condition; and
automatically stopping travel motion of the mobile work machine based on the detected stop condition.

### Claim 6 (depends on 5)

The computer implemented method of claim 5 wherein detecting a stop condition comprises:
detecting that the mobile work machine has reached an end of the route line.

### Claim 7 (depends on 4)

The computer implemented method of claim 4 wherein controlling the propulsion subsystem and the steering subsystem comprises:
detecting a desired distance interval; and
controlling the propulsion subsystem and the steering subsystem on the mobile work machine so the mobile work machine travels along the desired travel route for the desired distance interval.

### Claim 8 (depends on 7)

The computer implemented method of claim 7 wherein controlling the propulsion subsystem and the steering subsystem on the mobile work machine so the mobile work machine travels along the desired travel route for the desired distance interval comprises:
detecting that the mobile work machine has travelled along the travel route for the desired distance interval; and
stopping travel motion of the mobile work machine.

### Claim 9 (depends on 1)

The computer implemented method of claim 1 wherein identifying an operator-generated marking comprises:
identifying an exclusion zone indicating a geographic portion of the worksite that the mobile work machine is to avoid.

### Claim 10 (depends on 9)

The computer implemented method of claim 9 wherein automatically controlling machine travel comprises:
detecting a travel command input commanding the mobile work machine to perform a commanded operation;
determining whether the commanded operation will cause a periphery of the mobile work machine to intersect with the exclusion zone;
generating a likely intersection signal indicative of whether the commanded operation will cause a periphery of the mobile work machine to intersect with the exclusion zone; and
automatically controlling a controllable subsystem of the mobile work machine based on the likely intersection signal.

### Claim 11 (depends on 10)

The computer implemented method of claim 10 wherein, when the likely intersection signal indicates that the commanded operation will cause a periphery of the mobile work machine to intersect with the exclusion zone, then automatically controlling a controllable subsystem comprises:
controlling the controllable subsystem of the mobile work machine to inhibit performance of the commanded operation.

### Claim 12 (depends on 10)

The computer implemented method of claim 10 wherein, when the likely intersection signal indicates that the commanded operation will cause a periphery of the mobile work machine to intersect with the exclusion zone, then automatically controlling a controllable subsystem comprises:
controlling an operator interface mechanism to generate an operator alert.

### Claim 13 (depends on 1)

The computer implemented method of claim 1 wherein detecting the attachment control operator input comprises:
detecting the attachment control operator input from a remote operator.

### Claim 14

A mobile work machine, comprising:
an attachment controlled by an attachment control operator input;
a backup camera capturing an image of a worksite;
a mark identification system identifying an operator-generated marking, having a predefined characteristic, in the image;
a mark-to-machine localization system localizing the predefined operator-generated marking to the mobile work machine;
a control signal generator generating control signals to control operation of the attachment on the mobile work machine based on the detected attachment control operator input; and
a travel control system automatically controlling machine travel of the mobile work machine based on the localized, operator-generated marking.

### Claim 15 (depends on 14)

The mobile work machine of claim 14 wherein the mark identification system is configured to identify a route line indicating a desired travel route of the mobile work machine and wherein the travel control system comprises:
a commanded travel motion detector that detects a travel command input commanding the mobile work machine to travel; and
a fixed speed component configured to detect a target speed and controlling a propulsion subsystem and a steering subsystem on the mobile work machine so the mobile work machine travels along the desired travel route until at the target speed.

### Claim 16 (depends on 14)

The mobile work machine of claim 14 wherein the mark identification system is configured to identify a route line indicating a desired travel route of the mobile work machine and the travel control system comprises:
a desired distance detector detecting a desired distance interval; and
a fixed distance component controlling the propulsion subsystem and the steering subsystem on the mobile work machine so the mobile work machine travels along the desired travel route for the desired distance interval.

### Claim 17 (depends on 16)

The mobile work machine of claim 16 wherein the fixed distance component comprises:
a distance comparator detecting that the mobile work machine has travelled along the travel route for the desired distance interval; and
a stop condition detector generating a stop signal to stop travel motion of the mobile work machine.

### Claim 18 (depends on 14)

The mobile work machine of claim 14 wherein the mark identification system is configured to identify an exclusion zone indicating a geographic portion of the worksite that the mobile work machine is to avoid, wherein the travel control system comprises:
a commanded travel motion detector that detects a travel command input commanding the mobile work machine to perform a commanded operation; and
a commanded motion/exclusion zone analysis system that determines whether the commanded operation will cause a periphery of the mobile work machine to intersect with the exclusion zone and generate a likely intersection signal indicative of whether the commanded operation will cause a periphery of the mobile work machine to intersect with the exclusion zone, the travel control system being configured to automatically control a controllable subsystem of the mobile work machine based on the likely intersection signal.

### Claim 19

A computer implemented method of controlling a mobile work machine, comprising:
capturing an image of a worksite from a backup camera on the mobile work machine;
identifying an operator-generated marking, having a predefined characteristic, in the image, the operator-generated marking identifying a material transport route on the worksite;
localizing the material transport route indicated by the operator-generated markings to the mobile work machine;
detecting a machine travel operator input; and
automatically controlling the mobile work machine to travel along the material transport route by continuing to capture images from the backup camera, localizing the material transport route to the mobile work machine, and controlling a steering subsystem and a propulsion subsystem to follow the material transport route.

### Claim 20 (depends on 19)

The computer implemented method of claim 19 wherein detecting a machine travel operator input comprises:
detecting the machine travel operator input from a remote device, remote from an operator compartment of the mobile work machine.
