# US 2021/0174088 A1 · System and method for detecting objects

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174088 A1
- Document ID: US20210174088A1
- Status: Published application
- Kind: A1
- Application number: 16/709,150
- Filed: 2019-12-10
- Published: 2021-06-10
- Earliest claimed: 2019-12-10
- Assignees: Caterpillar Inc.
- Applicants: Caterpillar Inc.
- Inventors: Jacob Charles Maley; Pranay Kumar Reddy Kontham; Robert Scott McFarland; Vamsi Krishna Pannala; Peter Joseph Petrany
- Web page: https://patentgrep.com/patent/US20210174088A1

## CPC

- G06V 20/30: in albums, collections or shared content, e.g. social network photos or video
- G06N 20/00: Machine learning
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/64: Three-dimensional [3D] objects
- G06T 2207/10012: Stereo images
- G06T 2207/30261: Obstacle
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An object detection system may include an imager configured to generate image data indicative of an environment in which a machine is present, and a sensor configured to generate sensor data indicative of the environment in which the machine is present. The object detection system may further include an object detection controller including one or more object detection processors configured to receive an image signal indicative of the image data, identify an object associated with the image data, and determine a first location of the object relative to the position of the imager. The one or more object detection processors may also be configured to receive a sensor signal indicative of the sensor data, and determine, based at least in part on the sensor signal, the presence or absence of the object at the first location.

## Claims

### Claim 1

An object detection system comprising:
an imager configured to generate image data indicative of an environment in which a machine is present relative to a first position of the imager;
a sensor configured to generate sensor data indicative of the environment in which the machine is present relative to a second position of the sensor;
an object detection controller comprising one or more object detection processors configured to:
receive an image signal indicative of the image data;
identify an object associated with the image data;
determine a first location of the object relative to the first position of the imager;
receive a sensor signal indicative of the sensor data; and
determine, based at least in part on the sensor signal, a presence of the object at the first location.

### Claim 2 (depends on 1)

The system of claim 1, wherein the imager comprises a first imager and the sensor comprises a second imager.

### Claim 3 (depends on 1)

The system of claim 1, wherein determining the first location of the object relative to the first position of the imager comprises:
determining a first distance between the first position of the imager and the first location of the object; and
determining a first angle indicative of the first location of the object relative to the first position of the imager.

### Claim 4 (depends on 1)

The system of claim 1, wherein the one or more object detection processors are further configured to determine a projected location of the object relative to the second position of the sensor, wherein determining the projected location of the object relative to the second position of the sensor comprises transforming the first location of the object from a first frame of reference associated with the first position of the imager to a second frame of reference associated with the second position of the sensor.

### Claim 5 (depends on 4)

The system of claim 4, wherein determining the presence of the object at the first location comprises one of:
determining the presence of the object at the first location when the sensor signal indicative of the sensor data comprises sensor data indicative of the object at the projected location; or
determining an absence of the object at the first location when the sensor signal indicative of the sensor data lacks sensor data indicative of the object at the projected location.

### Claim 6 (depends on 1)

The system of claim 1, wherein the one or more object detection processors are further configured to substantially temporally align the image data and the sensor data.

### Claim 7 (depends on 1)

The system of claim 1, wherein identifying the object associated with the image data comprises executing an analytical model configured to determine, based at least in part on the image data, the presence of the object, wherein the analytical model comprises an object detection model developed by a machine learning engine.

### Claim 8 (depends on 7)

The system of claim 7, wherein the one or more object detection processors are further configured to generate training data for training the object detection model, the training data comprising object detection data associated with one of determining the presence of the object at the first location or an absence of the object at the first location.

### Claim 9 (depends on 1)

The system of claim 1, wherein first position of the imager and the second position of the sensor are separated from one another by a separation distance greater than about one foot.

### Claim 10

A method for detecting an object, the method comprising:
receiving an image signal indicative of image data from an imager;
identifying an object associated with the image data;
determining a first location of the object relative to a first position of the imager;
receiving a sensor signal indicative of sensor data generated by a sensor; and
determining, based at least in part on the sensor signal, one of a presence of the object at the first location or an absence of the object at the first location.

### Claim 11 (depends on 10)

The method of claim 10, wherein determining the first location of the object relative to the first position of the imager comprises:
determining a first distance between the first position of the imager and the first location of the object; and
determining a first angle indicative of the first location of the object relative to the first position of the imager.

### Claim 12 (depends on 10)

The method of claim 10, further comprising determining a projected location of the object relative to a second position of the sensor, wherein determining a projected location of the object relative to the second position of the sensor comprises transforming the first location of the object from a first frame of reference associated with the first position of the imager to a second frame of reference associated with the second position of the sensor.

### Claim 13 (depends on 12)

The method of claim 12, wherein determining one of a presence of the object at the first location or an absence of the object at the first location comprises one of:
determining the presence of the object at the first location when the sensor signal indicative of the sensor data comprises sensor data indicative of the object at the projected location; or
determining the absence of the object at the first location when the sensor signal indicative of the sensor data lacks sensor data indicative of the object at the projected location.

### Claim 14 (depends on 10)

The method of claim 10, further comprising substantially temporally aligning the image data and the sensor data.

### Claim 15 (depends on 10)

The method of claim 10, wherein identifying the object associated with the image data comprises executing an analytical model configured to determine, based at least in part on the image data, the presence of the object, wherein the analytical model comprises an object detection model developed by a machine learning engine.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
generating training data for training the object detection model using the machine learning engine, the training data comprising object detection data associated with determining one of the presence of the object at the first location or the absence of the object at the first location; and
training the object detection model using the training data.

### Claim 17

A machine comprising:
a chassis;
an imager mounted to the chassis at a first position of the chassis, the imager configured to generate image data indicative of an environment in which the machine is present relative to a first position of the imager;
a sensor mounted to the chassis at a second position of the chassis spaced from the first position of the chassis, the sensor configured to generate sensor data indicative of the environment in which the machine is present relative to the second position of the sensor; and
an object detection controller comprising one or more object detection processors configured to:
identify, based at least in part on the image data, an object associated with the image data;
determine a first location of the object relative to the first position of the imager;
receive a sensor signal indicative of the sensor data; and
determine, based at least in part on the sensor signal, one of a presence of the object at the first location or an absence of the object at the first location.

### Claim 18 (depends on 17)

The machine of claim 17, wherein first position of the imager and the second position of the sensor are separated from one another by a separation distance greater than about one foot.

### Claim 19 (depends on 17)

The machine of claim 17, wherein determining the first location of the object relative to the first position of the imager comprises:
determining a first distance between the first position of the imager and the first location of the object; and
determining a first angle indicative of the first location of the object relative to the first position of the imager.

### Claim 20 (depends on 17)

The machine of claim 17, wherein the one or more object detection processors are further configured to determine a projected location of the object relative to the second position of the sensor, wherein determining the projected location of the object relative to the second position of the sensor comprises transforming the first location of the object from a first frame of reference associated with the first position of the imager to a second frame of reference associated with the second position of the sensor.
