# US 11,113,548 B2 · Object detection neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,548 B2
- Document ID: US11113548B2
- Status: Granted
- Kind: B2
- Application number: 16/436,754
- Filed: 2019-06-10
- Published: 2019-09-26
- Granted: 2021-09-07
- Earliest claimed: 2016-12-19
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Zhaoyin Jia; Ury Zhilinsky; Yun Jiang; Yimeng Zhang
- Web page: https://patentgrep.com/patent/US11113548B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/25: Fusion techniques
- G06N 3/00: Computing arrangements based on biological models
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 10/82: using neural networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating object detection predictions from a neural network. In some implementations, an input characterizing a first region of an environment is obtained. The input includes a projected laser image generated from a three-dimensional laser sensor reading of the first region, a camera image patch generated from a camera image of the first region, and a feature vector of features characterizing the first region. The input is processed using a high precision object detection neural network to generate a respective object score for each object category in a first set of one or more object categories. Each object score represents a respective likelihood that an object belonging to the object category is located in the first region of the environment.

## Claims

### Claim 1

A method comprising:
processing, by one or more computers, a first input generated from first sensor data collected by one or more first sensors of a vehicle using a first high-recall object detection neural network, wherein the first sensor data characterizes an environment in a vicinity of the vehicle, and wherein the first high-recall object detection neural network is configured to:
receive the first input; and
process the first input to generate:
for each of a plurality of bounding boxes that each correspond to a respective region of the environment, a respective first confidence score that represents a probability that an object is present in the region of the environment corresponding to the bounding box;
selecting, based at least on the respective first confidence scores, a first region of the environment corresponding to a first bounding box of the plurality of bounding boxes;
in response to selecting the first region:
obtaining, by the one or more computers, a second input characterizing the first region of the environment, wherein the second input is different from the first input and is generated from second sensor data collected by a different, second set of sensors than the first input, the different, second set of sensors comprising the one or more first sensors and at least one additional sensor that is of a different type than that of the one or more first sensors; and
processing, by the one or more computers, the second input using a high precision object detection neural network to generate a respective object score for each object category in a set of one or more object categories, wherein each object score represents a respective probability that an object belonging to the object category is present in the first region of the environment.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more first sensors comprise a single camera sensor of the vehicle, and wherein the one additional sensor that is of a different type than that of the one or more first sensors comprises a laser sensor of the vehicle.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
processing, by one or more computers, a third input generated from third sensor data collected by one or more third sensors of the vehicle using a third high-recall object detection neural network, wherein the third sensor data characterizes the environment in the vicinity of the vehicle, and wherein the third high-recall object detection neural network is configured to:
receive the third input; and
process the third input to generate:
for each of the plurality of bounding boxes, a respective third confidence score that represents a probability that an object is present in the region of the environment that shows the bounding box.

### Claim 4 (depends on 3)

The method of claim 3, wherein selecting the first region is based on the first confidence score for the first bounding box and the third confidence score for the first bounding box.

### Claim 5 (depends on 3)

The method of claim 3, wherein the one or more first sensors are different from the one or more third sensors.

### Claim 6 (depends on 5)

The method of claim 5, wherein the second set of sensors include the first sensors and the third sensors.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first high-recall object detection neural network generates data identifying the plurality of bounding boxes.

### Claim 8 (depends on 1)

The method of claim 1, wherein generating the second input comprises:
generating, from the second sensor data, one or more projected laser images characterizing the first region of the environment.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the second input comprises:
generating, from the second sensor data, one or more camera image patches corresponding to the first bounding box for the first region of the environment.

### Claim 10 (depends on 1)

The method of claim 1, wherein generating the second input comprises:
generating a feature vector of features that comprises context information characterizing the first region of the environment.

### Claim 11

A system comprising:
one or more computers; and
a non-transitory computer-readable medium coupled to the one or more computers having instructions stored thereon, which, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
processing, by one or more computers, a first input generated from first sensor data collected by one or more first sensors of a vehicle using a first high-recall object detection neural network, wherein the first sensor data characterizes an environment in a vicinity of the vehicle, and wherein the first high-recall object detection neural network is configured to:
receive the first input; and
process the first input to generate:
for each of a plurality of bounding boxes that each correspond to a respective region of the environment, a respective first confidence score that represents a probability that an object is present in the region of the environment corresponding to the bounding box;
selecting, based at least on the respective first confidence scores a first region of the environment corresponding to a first bounding box of the plurality of bounding boxes;
in response to selecting the first region:
obtaining, by the one or more computers, a second input characterizing the first region of the environment, wherein the second input is different from the first input and is generated from the second sensor data collected by a different, second set of sensors than the first input, the different, second set of sensors comprising the one or more first sensors and at least one additional sensor that is of a different type than that of the one or more first sensors; and
processing, by the one or more computers, the second input using a high precision object detection neural network to generate a respective object score for each object category in a set of one or more object categories, wherein each object score represents a respective probability that an object belonging to the object category is present in the first region of the environment.

### Claim 12 (depends on 11)

The system of claim 11, wherein the one or more first sensors comprise a single camera sensor of the vehicle, and wherein the one additional sensor that is of a different type than that of the one or more first sensors comprises a laser sensor of the vehicle.

### Claim 13 (depends on 11)

The system of claim 11, further comprising:
processing, by one or more computers, a third input generated from third sensor data collected by one or more third sensors of the vehicle using a third high-recall object detection neural network, wherein the third sensor data characterizes the environment in the vicinity of the vehicle, and wherein the third high-recall object detection neural network is configured to:
receive the third input; and
process the third input to generate:
for each of the plurality of bounding boxes, a respective third confidence score that represents a probability that an object is present in the region of the environment that shows the bounding box.

### Claim 14 (depends on 13)

The system of claim 13, wherein selecting the first region is based on the first confidence score for the first bounding box and the third confidence score for the first bounding box.

### Claim 15 (depends on 13)

The system of claim 13, wherein the one or more first sensors are different from the one or more third sensors.

### Claim 16 (depends on 15)

The system of claim 15, wherein the second set of sensors include the first sensors and the third sensors.

### Claim 17 (depends on 11)

The system of claim 11, wherein the first high-recall object detection neural network generates data identifying the plurality of bounding boxes.

### Claim 18

A non-transitory computer storage device encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
processing, by one or more computers, a first input generated from first sensor data collected by one or more first sensors of a vehicle using a first high-recall object detection neural network, wherein the first sensor data characterizes an environment in a vicinity of the vehicle, and wherein the first high-recall object detection neural network is configured to:
receive the first input; and
process the first input to generate:
for each of a plurality of bounding boxes that each correspond to a respective region of the environment, a respective first confidence score that represents a probability that an object is present in the region of the environment corresponding to the bounding box;
selecting, based at least on the respective first confidence scores a first region of the environment corresponding to a first bounding box of the plurality of bounding boxes;
in response to selecting the first region:
obtaining, by the one or more computers, a second input characterizing the first region of the environment, wherein the second input is different from the first input and is generated from second sensor data collected by a different, second set of sensors than the first input, the different, second set of sensors comprising the one or more first sensors and at least one additional sensor that is of a different type than that of the one or more first sensors; and
processing, by the one or more computers, the second input using a high precision object detection neural network to generate a respective object score for each object category in a set of one or more object categories, wherein each object score represents a respective probability that an object belonging to the object category is present in the first region of the environment.