# US 12,183,088 B2 · Identifying objects using LiDAR

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,088 B2
- Document ID: US12183088B2
- Status: Granted
- Kind: B2
- Application number: 17/504,449
- Filed: 2021-10-18
- Published: 2022-04-21
- Granted: 2024-12-31
- Earliest claimed: 2020-10-21
- Assignees: Motional AD LLC
- Applicants: Motional AD LLC
- Inventors: Venice Erin Baylon Liong; Tho Thi Ngoc Nguyen
- Web page: https://patentgrep.com/patent/US12183088B2

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G01S 7/4802: {using analysis of echo signal for target characterisation; Target signature; Target cross-section}
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G05B 13/027: {using neural networks only}
- G06F 18/24: Classification techniques
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/809: {of classification results, e.g. where the classifiers operate on the same input data}
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Among other things, techniques are described for controlling, using a control circuit, motion of a vehicle based objects identified using LiDAR. For example, respective classes of points of a point cloud are determined, and based on the determined respective classes of the points of the point cloud, objects in the vicinity of the vehicle are identified.

## Claims

### Claim 1

A method, comprising:
receiving, using at least one processor, LiDAR point cloud data associated with a plurality of points in a point cloud;
generating, using the at least one processor, a Birds-eye View (BeV) of the point cloud and a Range View (RV) of the point cloud based on the plurality of points in the point cloud, wherein the RV is different from the BeV;
providing, using the at least one processor, the BeV as input to a first view neural network and the RV as input to a RV neural network, the RV neural network different from the BeV neural network;
generating, using the at least one processor, for each point in the point cloud, a first set of class scores indicative of object classes using the BeV neural network, and a second set of class scores indicative of the object classes using the RV neural network, wherein the first set of class scores and the second set of class scores are generated in parallel;
determining, using the at least one processor, a finalized label for at least one point in the point cloud, wherein the determining is based on the first set of class scores of the at least one point and the second set of class scores of the at least one point;
identifying, using the at least one processor, at least one object in a vicinity of a vehicle based at least in part on the finalized label of the at least one point; and
controlling, using the at least one processor, motion of the vehicle based on the at least one object.

### Claim 2 (depends on 1)

The method of claim 1, wherein the determining a finalized label for at least one point in the point cloud comprises:
determining at least one uncertain point in the point cloud, wherein the determining is based on the first set of class scores of the at least one uncertain point and the second set of class scores of the at least one uncertain point;
generating, using a fusion neural network, a third set of class scores for the at least one uncertain point based on the at least one of the first set of class scores of the at least one uncertain point and the second set of class scores of the at least one uncertain point, wherein the third set of class scores is based on characteristics of neighboring points of the at least one uncertain point; and
based on the third set of class scores, determining, using the at least one processor, a finalized label of the at least one uncertain point.

### Claim 3 (depends on 1)

The method of claim 1, wherein the LiDAR point cloud data comprises information about a color associated with at least one point included in the point cloud.

### Claim 4 (depends on 1)

The method of claim 1, wherein the LiDAR point cloud data comprises point intensity information.

### Claim 5 (depends on 2)

The method of claim 2, wherein at least one class score included in the first set of class scores, the second set of class scores or the third set of class scores of a particular point corresponds to a pre-defined class of object.

### Claim 6 (depends on 2)

The method of claim 2, wherein at least one of the BeV neural network, the RV neural network, and the fusion neural network comprises at least one recurrent layer.

### Claim 7 (depends on 1)

The method of claim 1, wherein providing the BeV data as input to the BeV neural network and the RV data as input to the RV neural network comprises:
providing the BeV data as input to the BeV neural network and the RV data as input to the RV neural network, the BeV data or the RV data generated at least in part based on data augmentation.

### Claim 8 (depends on 1)

The method of claim 1, wherein the uncertain point is determined with respect to a threshold difference in class scores, wherein the threshold difference is determined based on at least one of a probability function or a filtering function or both.

### Claim 9 (depends on 1)

The method of claim 1, wherein a class score for the at least one uncertain point is determined based on concatenated features of neighboring points of the at least one uncertain point.

### Claim 10 (depends on 9)

The method of claim 9, wherein the concatenated features comprise intermediate output of intermediate layers of the BeV neural network and the RV neural network.

### Claim 11 (depends on 9)

The method of claim 9, wherein the concatenated features comprise the output class scores from the at least one of the BeV neural network and the RV neural network.

### Claim 12 (depends on 2)

The method of claim 2, wherein the fusion neural network includes at least one of a multi-layer perceptron or a convolutional layer.

### Claim 13

A vehicle, comprising:
at least one LiDAR device capable of generating a LiDAR scan point cloud that includes a plurality of LiDAR data points; and
a processing circuit coupled to the LiDAR device, the processing circuit configured for:
receiving LiDAR point cloud data associated with a plurality of points in a point cloud;
generating a Birds-eye View (BeV) of the point cloud and a Range View (RV) of the point cloud based on the plurality of points in the point cloud, wherein the RV is different from the BeV;
providing the BeV as input to a BeV neural network and the RV as input to a RV neural network, the RV neural network different from the BeV neural network;
generating for each point in the point cloud, a first set of class scores indicative of object classes using the BeV neural network, and a second set of class scores indicative of the object classes using the RV neural network, wherein the first set of class scores and the second set of class scores are generated in parallel;
determining a finalized label for at least one point in the point cloud, wherein the determining is based on the first set of class scores of the at least one point and the second set of class scores of the at least one point;
identifying at least one object in a vicinity of a vehicle based at least in part on the finalized label of the at least one point; and
controlling motion of the vehicle based on the at least one object.

### Claim 14 (depends on 13)

The vehicle of claim 13, wherein determining a finalized label for at least one point in the point cloud comprises:
determining at least one uncertain point in the point cloud, wherein the determining is based on the first set of class scores of the at least one uncertain point and the second set of class scores of the at least one uncertain point;
generating, using a fusion neural network, a third set of class scores for the at least one uncertain point based on the at least one of the first set of class scores of the at least one uncertain point and the second set of class scores of the at least one uncertain point, wherein the third set of class scores is based on characteristics of neighboring points of the at least one uncertain point; and
based on the third set of class scores, determining a finalized label of the at least one uncertain point.

### Claim 15 (depends on 13)

The vehicle of claim 13, wherein the LiDAR point cloud data comprises information about a color associated with at least one point included in the point cloud.

### Claim 16 (depends on 13)

The vehicle of claim 13, wherein the LiDAR point cloud data comprises point intensity information.

### Claim 17 (depends on 14)

The vehicle of claim 14, wherein at least one class score included in the first set of class scores, the second set of class scores or the third set of class scores of a particular point corresponds to a pre-defined class of object.

### Claim 18 (depends on 14)

The vehicle of claim 14, wherein at least one of the BeV neural network, the RV neural network, and the fusion neural network comprises at least one recurrent layer.

### Claim 19 (depends on 13)

The vehicle of claim 13, wherein providing the BeV data as input to the BeV neural network and the RV data as input to the RV neural network comprises:
providing the BeV data as input to the BeV neural network and the RV data as input to the RV neural network, the BeV data or the RV data generated at least in part based on data augmentation.

### Claim 20 (depends on 13)

The vehicle of claim 13, wherein the uncertain point is determined with respect to a threshold difference in class scores, wherein the threshold difference is determined based on at least one of a probability function or a filtering function or both.

### Claim 21 (depends on 13)

The vehicle of claim 13, wherein a class score for the at least one uncertain point is determined based on concatenated features of neighboring points of the at least one uncertain point.

### Claim 22 (depends on 21)

The vehicle of claim 21, wherein the concatenated features comprise intermediate output of intermediate layers of the BeV neural network and the RV neural network.

### Claim 23 (depends on 21)

The vehicle of claim 21, wherein the concatenated features comprise the output class scores from the BeV neural network and the RV neural network.

### Claim 24 (depends on 14)

The vehicle of claim 14, wherein the fusion neural network includes at least one of a multi-layer perceptron or a convolutional layer.