# US 2022/0044068 A1 · Processing perspective view range images using neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044068 A1
- Document ID: US20220044068A1
- Status: Published application
- Kind: A1
- Application number: 17/443,674
- Filed: 2021-07-27
- Published: 2022-02-10
- Earliest claimed: 2020-07-27
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Yuning Chai; Pei Sun; Jiquan Ngiam; Weiyue Wang; Vijay Vasudevan; Benjamin James Caine; Xiao Zhang; Dragomir Anguelov
- Web page: https://patentgrep.com/patent/US20220044068A1

## CPC

- G06F 18/27: Regression, e.g. linear or logistic regression
- G01S 7/4802: {using analysis of echo signal for target characterisation; Target signature; Target cross-section}
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 17/89: for mapping or imaging
- G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/253: {of extracted features}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 3/4046: using neural networks
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- H04N 23/10: for generating image signals from different wavelengths
- G06T 2207/20084: Artificial neural networks [ANN]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for processing a perspective view range image generated from sensor measurements of an environment. The perspective view range image includes a plurality of pixels arranged in a two-dimensional grid and including, for each pixel, (i) features of one or more sensor measurements at a location in the environment corresponding to the pixel and (ii) geometry information comprising range features characterizing a range of the location in the environment corresponding to the pixel relative to the one or more sensors. The system processes the perspective view range image using a first neural network to generate an output feature representation. The first neural network comprises a first perspective point-set aggregation layer comprising a geometry-dependent kernel.

## Claims

### Claim 1

A method performed by one or more computers, the method comprising:
obtaining a perspective view range image generated from sensor measurements of an environment by one or more sensors, the perspective view range image comprising a plurality of pixels arranged in a two-dimensional grid and including, for each pixel, (i) features of one or more sensor measurements at a location in the environment corresponding to the pixel and (ii) geometry information comprising range features characterizing a range of the location in the environment corresponding to the pixel relative to the one or more sensors;
processing the perspective view range image using a first neural network to generate an output feature representation, wherein the first neural network comprises a first perspective point-set aggregation layer configured to:
receive an input feature map, the input feature map comprising a respective feature vector for each of a first subset of the pixels; and
generate an output feature map from the input feature map, wherein the output feature map comprises a respective output feature vector for each of the first subset of pixels, and wherein the generating comprises, for each particular pixel in the first subset, generating an initial output feature vector for the particular pixel by applying a geometry-dependent kernel to pixels within a local neighborhood of the particular pixel in the input feature map, wherein the geometry-dependent kernel depends on at least (i) respective input feature vectors for the pixels within the local neighborhood of the particular pixel in the input feature map and (ii) respective range features of the pixels within the local neighborhood of the input feature map; and
processing the output feature representation using an output neural network to generate a network output for a neural network task.

### Claim 2 (depends on 1)

The method of claim 1, wherein the neural network task is object detection and the network output identifies portions of the environment where objects are located.

### Claim 3 (depends on 1)

The method of claim 1 wherein the perspective view range image is generated from sensor measurements from a LiDAR sensor sweeping through the environment, wherein one dimension of the two-dimensional grid corresponds to beams of the LiDAR sensor and wherein the other dimension of the two-dimensional grid corresponds to regions of the environment swept through by the LiDAR sensor.

### Claim 4 (depends on 1)

The method of claim 1, wherein the perspective view range image is generated from sensor measurements from an RGBD camera.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first neural network further comprises a two-dimensional convolutional layer that has a kernel that depends only on feature vectors and not on range features.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
obtaining validity data that indicates, for each pixel in the range image, whether the sensor measurements for the pixel are valid.

### Claim 7 (depends on 6)

The method of claim 6, wherein the geometry-dependent kernel also depends on, for each pixel in the local neighborhood, whether the sensor measurements for the pixel are valid.

### Claim 8 (depends on 7)

The method of claim 7, wherein the geometry-dependent kernel assigns the initial output feature vector for each pixel for which the sensor measurements are not valid to a zero vector.

### Claim 9 (depends on 7)

The method of claim 7, wherein the geometry-dependent kernel is a geometry-dependent convolution kernel that, when generating the initial output feature vector for each particular pixel for which the sensor measurements are valid, applies different convolution weights to input feature vectors of pixels depending on a range difference between the pixel and the particular pixel as reflected by the geometry information.

### Claim 10 (depends on 9)

The method of claim 9, wherein the geometry-dependent kernel has k sets of convolution weights, wherein each of the k sets of convolution weights has a respective scalar range, and wherein the convolution weights that are applied to input feature vectors of a given pixel for which the sensor measurements are valid are a combination of sets of convolution weights having respective scalar ranges that are satisfied by the range difference between the pixel and the particular pixel.

### Claim 11 (depends on 7)

The method of claim 7, wherein:
the geometry-dependent kernel is a self-attention kernel that applies a self-attention mechanism over the local neighborhood using queries, keys, and values for the pixels in the local neighborhood that are generated from the input feature vectors for the pixels in the local neighborhood, and
for each pixel in the local neighborhood for which the sensor measurements are valid, at least the key for the pixel is augmented with a positional encoding that represents a relative location of the pixel to the particular pixel as reflected by the geometry information.

### Claim 12 (depends on 7)

The method of claim 7, wherein the geometry-dependent kernel is a kernel that:
for each pixel in the local neighborhood for each particular pixel for which the sensor measurements are valid:
for each pixel in the local neighborhood for which the sensor measurements are valid, processes (i) the input feature vector for the pixel in the local neighborhood and (ii) a positional encoding that represents a relative location of the pixel to the particular pixel as reflected by the respective geometry information for the pixel and the particular pixel using an encoder neural network to generate encoded features for the pixel; and
applies max-pooling to the encoded features for the pixels in the local neighborhood to generate the initial output feature vector for the particular pixel.

### Claim 13 (depends on 7)

The method of claim 7, wherein the geometry-dependent kernel is a kernel that:
for each pixel in the local neighborhood for each particular pixel for which the sensor measurements are valid:
for each pixel in the local neighborhood for which the sensor measurements are valid, processes (i) the input feature vector for the pixel in the local neighborhood, (ii) a positional encoding that represents a relative location of the pixel to the particular pixel as reflected by the respective geometry information for the pixel and the particular pixel, and (iii) the input feature vector for the particular pixel using an encoder neural network to generate encoded features for the pixel; and
applies max-pooling to the encoded features for the pixels in the local neighborhood to generate the initial output feature vector for the particular pixel.

### Claim 14 (depends on 7)

The method of claim 7, wherein the first neural network further comprises one or more down-sampling layers that down-sample an input feature map based in part on the validity data.

### Claim 15 (depends on 14)

The method of claim 14, wherein the first neural network further comprises one or more up-sampling layers.

### Claim 16 (depends on 1)

The method of claim 1, further comprising generating the input feature map by fusing camera image features with (i) features from the perspective view range image, (ii) output features generated by another neural network layer in the first neural network, or both.

### Claim 17

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform:
obtaining a perspective view range image generated from sensor measurements of an environment by one or more sensors, the perspective view range image comprising a plurality of pixels arranged in a two-dimensional grid and including, for each pixel, (i) features of one or more sensor measurements at a location in the environment corresponding to the pixel and (ii) geometry information comprising range features characterizing a range of the location in the environment corresponding to the pixel relative to the one or more sensors;
processing the perspective view range image using a first neural network to generate an output feature representation, wherein the first neural network comprises a first perspective point-set aggregation layer configured to:
receive an input feature map, the input feature map comprising a respective feature vector for each of a first subset of the pixels; and
generate an output feature map from the input feature map, wherein the output feature map comprises a respective output feature vector for each of the first subset of pixels, and wherein the generating comprises, for each particular pixel in the first subset, generating an initial output feature vector for the particular pixel by applying a geometry-dependent kernel to pixels within a local neighborhood of the particular pixel in the input feature map, wherein the geometry-dependent kernel depends on at least (i) respective input feature vectors for the pixels within the local neighborhood of the particular pixel in the input feature map and (ii) respective range features of the pixels within the local neighborhood of the input feature map; and
processing the output feature representation using an output neural network to generate a network output for a neural network task.

### Claim 18 (depends on 17)

The system of claim 17, wherein:
the stored instructions further cause the one or more computers to perform: obtaining validity data that indicates, for each pixel in the range image, whether the sensor measurements for the pixel are valid; and
the geometry-dependent kernel also depends on, for each pixel in the local neighborhood, whether the sensor measurements for the pixel are valid.

### Claim 19

One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform:
obtaining a perspective view range image generated from sensor measurements of an environment by one or more sensors, the perspective view range image comprising a plurality of pixels arranged in a two-dimensional grid and including, for each pixel, (i) features of one or more sensor measurements at a location in the environment corresponding to the pixel and (ii) geometry information comprising range features characterizing a range of the location in the environment corresponding to the pixel relative to the one or more sensors;
processing the perspective view range image using a first neural network to generate an output feature representation, wherein the first neural network comprises a first perspective point-set aggregation layer configured to:
receive an input feature map, the input feature map comprising a respective feature vector for each of a first subset of the pixels; and
generate an output feature map from the input feature map, wherein the output feature map comprises a respective output feature vector for each of the first subset of pixels, and wherein the generating comprises, for each particular pixel in the first subset, generating an initial output feature vector for the particular pixel by applying a geometry-dependent kernel to pixels within a local neighborhood of the particular pixel in the input feature map, wherein the geometry-dependent kernel depends on at least (i) respective input feature vectors for the pixels within the local neighborhood of the particular pixel in the input feature map and (ii) respective range features of the pixels within the local neighborhood of the input feature map; and
processing the output feature representation using an output neural network to generate a network output for a neural network task.

### Claim 20 (depends on 19)

The one or more computer storage media of claim 19, wherein:
the stored instructions further cause the one or more computers to perform: obtaining validity data that indicates, for each pixel in the range image, whether the sensor measurement for the pixel is valid; and
the first neural network further comprises one or more down-sampling layers that down-sample an input feature map based in part on the validity data.
