# US 12,110,660 B2 · Work machine 3D exclusion zone

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,110,660 B2
- Document ID: US12110660B2
- Status: Granted
- Kind: B2
- Application number: 17/679,680
- Filed: 2022-02-24
- Published: 2023-08-24
- Granted: 2024-10-08
- Earliest claimed: 2022-02-24
- Assignees: Caterpillar Inc.
- Applicants: Caterpillar Inc.
- Inventors: Christopher Wright; Justin Steinlage
- Web page: https://patentgrep.com/patent/US12110660B2

## CPC

- E02F 9/2033: {Limiting the movement of frames or implements, e.g. to avoid collision between implements and the cabin (sub-groups of E02F3/431 of E02F3/435 take precedence; for turntables E02F9/123)}
- E02F 9/2029: {Controlling the position of implements in function of its load, e.g. modifying the attitude of implements in accordance to vehicle speed (control for hydraulic or pneumatic drives E02F9/2203, E02F9/2221 and E02F9/2253 take precedence)}
- E02F 9/2054: {Fleet management}
- E02F 9/2253: {Controlling the travelling speed of vehicles, e.g. adjusting travelling speed according to implement loads, control of hydrostatic transmission}
- E02F 9/24: Safety devices {, e.g. for preventing overload (E02F9/226 takes precedence)}
- E02F 9/262: {with follow-up actions to control the work tool, e.g. controller}
- E02F 9/265: {with follow-up actions (e.g. control signals sent to actuate the work tool)}
- G06T 7/73: using feature-based methods
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 30/1444: {Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields}
- G06T 2207/10012: Stereo images
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- E02F: DREDGING; SOIL-SHIFTING (winning peat E21C49/00)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A work machine, a first method of defining a virtual 3D exclusion zone, and a second method of preventing collisions involving a work machine are disclosed. The work machine comprises a body, an implement arm, an imaging assembly, and an electrohydraulic assembly configured to prevent the implement arm from intersecting a 3D exclusion zone. The first method comprises scanning a local environment, generating a virtual 3D representation, identifying key structures, and generating a virtual 3D exclusion zone encompassing the key structures. The second method comprises defining a virtual 3D exclusion zone, monitoring a motion of the work machine, and adjusting the motion to avoid an intersection between the work machine and the 3D exclusion zone. The 3D exclusion zone may be implemented for a number of machines, environments, and key structures without unduly removing control from a human operator.

## Claims

### Claim 1

A method of defining a virtual 3D exclusion zone, comprising the steps of:
scanning a local environment of a first work machine;
identifying a second work machine within the local environment;
generating a virtual 3D representation of the local environment and the second work machine;
identifying one or more key structures of the second work machine;
orienting the first work machine with respect to the local environment;
generating a virtual 3D exclusion zone encompassing the key structures, in which the 3D exclusion zone defines a space that should not be occupied by the first work machine; and
redirecting a direction of a velocity vector of an implement arm of the first work machine or redirecting a direction of an angular velocity vector of the implement arm of the first work machine to avoid intersection of the implement arm with the 3D exclusion zone.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the first work machine consists of a loading machine, the second work machine consists of a transport vehicle, and the key structures of the second work machine include a dump bed and a cabin.

### Claim 3 (depends on 1)

The method according to claim 1, the step of scanning the local environment further including:
capturing a plurality of images using one or more stereo cameras located on the first work machine.

### Claim 4 (depends on 3)

The method according to claim 3, the step of identifying the second work machine further including:
recognizing a fiducial marking of the second work machine; and/or
processing the plurality of images through a first neural network configured to identify a model of the second work machine.

### Claim 5 (depends on 3)

The method according to claim 3, the step of generating the virtual 3D representation further including:
building a plurality of disparity maps from the plurality of images; and/or
building a 3D point cloud representative of the local environment and the second work machine.

### Claim 6 (depends on 3)

The method according to claim 3, the step of identifying the key structures further including:
acquiring the key structures from a preexisting work machine model; and/or
processing the plurality of images through a second neural network configured to identify the key structures of the second work machine.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising the steps of:
merging the 3D representation of the local environment with a preexisting environment model; and/or
merging the 3D representation of the second work machine with a preexisting work machine model.

### Claim 8 (depends on 7)

The method according to claim 7, the step of generating the virtual 3D exclusion zone further including:
defining the key structures as a plurality of exclusion points within the 3D point cloud;
defining a volume of a 3D exclusion zone based on the plurality of exclusion points and a tolerance margin; and
defining a deceleration zone surrounding the 3D exclusion zone, wherein the first work machine is decelerated in the deceleration zone.

### Claim 9 (depends on 1)

The method according to claim 1, wherein each step is continuously repeated while the first work machine is being operated.

### Claim 10

A method of preventing collisions involving a work machine through application of a virtual 3D exclusion zone, comprising the steps of:
orienting the work machine with respect to a local environment;
defining a virtual 3D exclusion zone;
monitoring a motion of the work machine, the motion includes a velocity vector of an implement arm of the work machine or an angular velocity vector of the implement arm of the work machine;
predicting if the motion will result in an intersection between the work machine and the 3D exclusion zone; and
if an intersection is predicted, adjusting the motion to avoid the intersection, in which the motion is adjusted by redirecting a direction of the velocity vector or redirecting a direction of the angular velocity vector.

### Claim 11 (depends on 10)

The method according to claim 10, wherein the motion further includes a position, a displacement vector and/or an angular displacement vector, and an acceleration vector and/or and an angular velocity vector of the implement arm of the work machine.

### Claim 12 (depends on 11)

The method according to claim 11, the step of adjusting the motion further including:
decelerating a magnitude of the velocity vector; and/or
decelerating a magnitude of the angular velocity vector.

### Claim 13 (depends on 11)

The method according to claim 11, further comprising the steps of:
defining a virtual deceleration zone surrounding the 3D exclusion zone;
detecting an intersection between the work machine and the deceleration zone; and
if the work machine intersects the deceleration zone, decelerating a magnitude of the velocity vector normal to a surface of the exclusion zone; and/or
if the work machine intersects the deceleration zone, decelerating a magnitude of the angular velocity vector parallel to the surface of the exclusion zone.

### Claim 14 (depends on 10)

The method according to claim 10, wherein the 3D exclusion zone includes a first volume encompassing a dump bed of a transport vehicle and a second volume encompassing a cabin and a space above the cabin of the transport vehicle.

### Claim 15 (depends on 10)

The method according to claim 10, wherein each step is continuously repeated while the work machine is being operated.

### Claim 16

A work machine comprising:
a body;
an implement arm supported by the body;
an imaging assembly configured to define a 3D exclusion zone, including:
one or more 3D imaging devices supported by the implement arm; and
an image processing unit;
a power unit; and
an electrohydraulic assembly operatively powered by the power unit, including:
one or more implement actuators configured to control a motion of the implement arm, the motion includes a velocity vector of the implement arm of the work machine or an angular velocity vector of the implement arm of the work machine;
one or more implement sensors configured to monitor the motion of the implement arm; and
an electrohydraulic control unit (ECU) configured to command the implement actuators and prevent the implement arm from intersecting the 3D exclusion zone, the intersection is prevented by the electrohydraulic control unit (ECU) redirecting a direction of the velocity vector or redirecting a direction of the angular velocity vector.

### Claim 17 (depends on 16)

The work machine according to claim 16, wherein the implement arm further includes:
a boom connected to the body; and
a stick connected to the boom;
wherein the imaging assembly further includes:
a pair of stereo cameras located on a left side and a right side of the boom; and/or
a pair of stereo cameras located on a left side and a right side of the stick.

### Claim 18 (depends on 16)

The work machine according to claim 16, wherein the 3D exclusion zone encompasses a dump bed, a cabin, and a space above the cabin of a transport vehicle.

### Claim 19 (depends on 16)

The work machine according to claim 16, further comprising:
a traction device operatively powered by the power unit;
a positioning unit configured to locate the work machine with respect to a global environment; and
a communication unit.