# US 2019/0318206 A1 · Training Machine Learning Model Based On Training Instances With: Training Instance Input Based On Autonomous Vehicle Sensor Data, and Training Instance Output Based On Additional Vehicle Sensor Data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318206 A1
- Document ID: US20190318206A1
- Status: Published application
- Kind: A1
- Application number: 16/173,669
- Filed: 2018-10-29
- Published: 2019-10-17
- Earliest claimed: 2018-04-11
- Assignees: Aurora Innovation, Inc.
- Applicants: Aurora Innovation, Inc.
- Inventors: Warren Smith; Ethan Eade; Sterling J. Anderson; James Andrew Bagnell; Bartholomeus C. Nabbe; Christopher Paul Urmson
- Web page: https://patentgrep.com/patent/US20190318206A1

## CPC

- B60W 40/04: Traffic conditions
- B60W 30/0956: {the prediction being responsive to traffic or environmental parameters}
- B60W 60/001: {Planning or execution of driving tasks}
- G01S 7/4808: {Evaluating distance, position or velocity data}
- G01S 17/34: using transmission of continuous, frequency-modulated waves while heterodyning the received signal, or a signal derived therefrom, with a locally-generated signal related to the contemporaneously transmitted signal
- G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46)
- G01S 17/58: Velocity or trajectory determination systems; Sense-of-movement determination systems
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- G01S 17/89: for mapping or imaging
- G01S 17/931: of land vehicles
- G05D 1/024: {in combination with a laser (lasers per se H01S)}
- G05D 1/247: using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2178: {based on feedback of a supervisor}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- B60W 2420/408: Radar; Laser, e.g. lidar
- B60W 2554/40: Dynamic objects, e.g. animals, windblown objects
- B60W 2554/4041: Position
- B60W 2554/803: Relative lateral speed
- B60W 2754/00: Output or target parameters relating to objects
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Various implementations described herein generate training instances that each include corresponding training instance input that is based on corresponding sensor data of a corresponding autonomous vehicle, and that include corresponding training instance output that is based on corresponding sensor data of a corresponding additional vehicle, where the corresponding additional vehicle is captured at least in part by the corresponding sensor data of the corresponding autonomous vehicle. Various implementations train a machine learning model based on such training instances. Once trained, the machine learning model can enable processing, using the machine learning model, of sensor data from a given autonomous vehicle to predict one or more properties of a given additional vehicle that is captured at least in part by the sensor data.

## Claims

### Claim 1

A method of training a machine learning model to be used in autonomous control of at least one autonomous vehicle, the method comprising:
generating a plurality of training instances, wherein generating each of the training instances includes:
generating training instance input of the training instance based on a corresponding instance of vision data from a vision component of a corresponding autonomous vehicle; and
generating a supervised training instance output of the training instance based on a corresponding instance of additional vehicle data that is based on one or more sensors of a corresponding additional vehicle that is captured by the corresponding instance of vision data, wherein the corresponding instance of additional vehicle data indicates a corresponding current state of at least one dynamic property of the corresponding additional vehicle;
wherein the corresponding instance of additional vehicle data is used in generating the supervised training instance output based on determining that the corresponding instance of additional vehicle data temporally corresponds to the corresponding instance of vision data;
training the machine learning model based on the plurality of training instances; and
providing the trained machine learning model for use in control of the at least one autonomous vehicle.

### Claim 2 (depends on 1)

The method of claim 1, wherein the corresponding instance of vision data is light Detection and Ranging (LIDAR) data and the vision component is a LIDAR component.

### Claim 3 (depends on 2)

The method of claim 2, wherein the LIDAR component is a phase coherent LIDAR component and the LIDAR data comprises a group of LIDAR data points of a sensing cycle of the phase coherent LIDAR component, each of the LIDAR data points of the group indicating a corresponding range and a corresponding velocity for a corresponding point in the environment, and each being generated based on a corresponding sensing event of the sensing cycle.

### Claim 4 (depends on 1)

The method of claim 1, wherein the supervised training instance output of the training instance further includes an indication of a bounding area that bounds a portion of the vision data that captures the corresponding additional vehicle.

### Claim 5 (depends on 1)

The method of claim 1, wherein training the machine learning model comprises:
processing, using the machine learning model, the training instance inputs of the training instances to generate predicted outputs;
generating losses based on the predicted outputs and the supervised training instance outputs of the training instances; and
updating the machine learning model based on the generated losses.

### Claim 6 (depends on 1)

The method of claim 1, wherein the instance of additional vehicle data comprises one or more yaw parameters and the one or more sensors of the corresponding additional vehicle comprise at least a yaw rate sensor.

### Claim 7 (depends on 6)

The method of claim 6, wherein the additional vehicle data is determined based on monitoring of a controller area network of the corresponding additional vehicle.

### Claim 8 (depends on 1)

The method of claim 1, wherein the instance of additional vehicle data comprises one or both of velocity and acceleration of the corresponding additional vehicle.

### Claim 9 (depends on 1)

The method of claim 1, wherein the instance of additional vehicle data comprises yaw parameters and one or both of velocity and acceleration.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
processing, by at least one processor of a given autonomous vehicle of the at least one autonomous vehicle, given vision data using the trained machine learning model, the given vision data captured by a given vision component of the given autonomous vehicle;
generating, based on the processing, a predicted state of a given additional vehicle captured by the given vision data; and
controlling the given autonomous vehicle based on the predicted state.

### Claim 11 (depends on 10)

The method of claim 10, wherein the given vision data is a subgroup of vision data captured during a sensing cycle of the vision component, and further comprising:
generating the subgroup of vision data based on determining the subgroup corresponds to the given additional vehicle.

### Claim 12 (depends on 11)

The method of claim 11, wherein determining the subgroup corresponds to the given additional vehicle comprises:
processing the vision data using an additional object detection and classification model; and
determining the subgroup corresponds to the given additional vehicle based on output generated based on processing the vision data using the additional object detection and classification model.

### Claim 13

A method of controlling an autonomous vehicle using a trained machine learning model, the method implemented by one or more processors of the autonomous vehicle, and the method comprising:
processing given vision data using the trained machine learning model, the given vision data captured by a vision component of the autonomous vehicle;
generating, based on the processing, a predicted state of at least one dynamic property of a given additional vehicle captured by the given vision data; and
controlling the autonomous vehicle based on the predicted state;
wherein the trained machine learning model is trained based on a plurality of training instances that each includes:
training instance input that is based on a corresponding instance of vision data from a vision component of a corresponding autonomous vehicle; and
supervised training instance output that is based on a corresponding instance of additional vehicle data that is based on one or more sensors of a corresponding additional vehicle that is captured by the corresponding instance of vision data.

### Claim 14 (depends on 13)

The method of claim 13, wherein the given vision data is a subgroup of vision data captured during a sensing cycle of the vision component, and further comprising:
generating the subgroup of vision data based on determining the subgroup corresponds to the given additional vehicle.

### Claim 15 (depends on 13)

The method of claim 13, wherein determining the subgroup corresponds to the given additional vehicle comprises:
processing the vision data using an additional object detection and classification model; and
determining the subgroup corresponds to the given additional vehicle based on output generated based on processing the vision data using the additional object detection and classification model.

### Claim 16 (depends on 13)

The method of claim 13, wherein controlling the autonomous vehicle comprises controlling speed and/or direction of the autonomous vehicle.

### Claim 17 (depends on 13)

The method of claim 13, wherein the at least one property comprises one or both of a yaw rate and a yaw direction of the additional vehicle.

### Claim 18 (depends on 13)

The method of claim 13, wherein the at least one property comprises one or both of velocity and acceleration of the additional vehicle.

### Claim 19 (depends on 13)

The method of claim 13, wherein the at least one property comprises a yaw parameter and one or both of velocity and acceleration of the additional vehicle.