# US 2019/0294177 A1 · Data augmentation using computer simulated objects for autonomous control systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0294177 A1
- Document ID: US20190294177A1
- Status: Published application
- Kind: A1
- Application number: 16/358,168
- Filed: 2019-03-19
- Published: 2019-09-26
- Earliest claimed: 2018-03-20
- Assignees: Phantom AI, Inc.
- Applicants: Phantom AI, Inc.
- Inventors: Youngwook Paul Kwon
- Web page: https://patentgrep.com/patent/US20190294177A1

## CPC

- G05D 1/0251: {extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision (stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593)}
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G06F 30/20: Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30)
- G06N 20/00: Machine learning
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A modeling system trains computer models for an autonomous control system using computer simulated models of objects. The objects may be vehicles, and the computer simulated models may be virtual models of vehicles simulated by computer software. Since the vehicle models are computer simulated, various characteristics of the vehicle can be easily obtained by the modeling system. The various types of data may include geometric information of the vehicle, views of the vehicle from different perspectives, and the like. The modeling system can easily generate and label a large amount of training data using the characteristics of the computer simulated vehicles. The modeling system can use the training data to train computer models for the autonomous control system.

## Claims

### Claim 1

A computer-implemented method of training a machine-learned transformation model configured to receive an image of a vehicle from a reference viewpoint and output a target image of the vehicle from a target viewpoint different from the reference viewpoint, comprising:
obtaining a training dataset of images that represent a set of computer simulated vehicles with a known set of characteristics, the training dataset including a training set of reference images that capture the computer simulated vehicles from one or more reference viewpoints and a corresponding training set of target images that capture the computer simulated vehicles from one or more target viewpoints different from the one or more reference viewpoints; and
training a set of parameters of the transformation model using the training dataset of images, the training comprising repeatedly performing iterations of:
generating a set of estimated target images by applying the transformation model with an estimated set of parameters to the training set of reference images,
determining a loss function indicating a difference between the set of estimated target images and the training set of target images, and
updating the set of parameters of the transformation model to reduce the loss function.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the set of computer simulated vehicles are computer-aided (CAD) generated models of vehicles, or computer simulated light detection and range (LIDAR) models of vehicles.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
obtaining a new image of a vehicle in a scene and a desired target viewpoint of the vehicle;
generating a predicted target image by applying the transformation model to a region of the new image containing the vehicle, the predicted target image predicting appearance of the vehicle from the desired target viewpoint; and
determining whether the vehicle is present in the region of the new image using the predicted target image.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the transformation model is further configured to a label indicating a desired target viewpoint, wherein the training dataset further includes a training set of labels indicating the one or more target viewpoints of the training set of target images, and wherein generating the set of estimated target images further comprises applying the transformation model with the estimated set of parameters to the training set of labels.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein a label in the training set of labels is represented as a set of coordinates or a set of categories of viewpoints.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more reference viewpoints are viewpoints showing side parts of the computer simulated vehicles.

### Claim 7

A computer-implemented method of training an object detection model configured to receive an image of a vehicle and output a set of 3-D bounding box coordinates of the vehicle in the image, comprising:
obtaining a training dataset of images that represent a set of computer simulated vehicles with a known set of characteristics, the characteristics including geometric information of the vehicles in the images;
obtaining a training set of coordinates for the images that represent 3-D bounding boxes of the computer simulated vehicles, the 3-D bounding boxes enclosing an outer boundary of the computer simulated vehicles, and the training set of coordinates for the 3-D bounding boxes determined based on the geometric information of the computer simulated vehicles; and
training a set of parameters of the object detection model using the training dataset of images, the training comprising repeatedly performing iterations of:
generating a set of estimated coordinates by applying the object detection model with an estimated set of parameters to the training dataset of images,
determining a loss function indicating a difference between the set of estimated coordinates and the training set of coordinates for the computer simulated vehicles, and
updating the set of parameters of the object detection model to reduce the loss function.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the set of computer simulated vehicles are computer-aided (CAD) generated models of vehicles, or computer simulated light detection and ranging (LIDAR) models of vehicles.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, further comprising:
obtaining a training set of orientations for the computer simulated vehicles that represent displacement of the computer simulated vehicles with respect to a reference axis,
wherein training the set of parameters further comprises:
generating a set of estimated orientations by applying the object detection model with the estimated set of parameters to the training dataset of images, and
wherein the loss function further indicates a difference between the set of estimated orientations and the training set of orientations for the computer simulated vehicles.

### Claim 10 (depends on 7)

The computer-implemented method of claim 7, further comprising:
obtaining a training set of categories for the computer simulated vehicles that represent classifications of the computer simulated vehicles with respect to their structure or functions,
wherein training the set of parameters further comprises:
generating a set of estimated categories by applying the object detection model with the estimated set of parameters to the training dataset of images, and
wherein the loss function further indicates a difference between the set of estimated categories and the training set of categories for the computer simulated vehicles.

### Claim 11 (depends on 7)

The computer-implemented method of claim 7, further comprising:
obtaining a new image of a vehicle in a scene;
generating a predicted set of coordinates of the vehicle by applying the object detection model to the new image, the predicted set of coordinates representing an estimated 3-D bounding box enclosing the vehicle; and
generating a transformed set of coordinates from the predicted set of coordinates, the transformed set of coordinates representing a transformed 3-D bounding box of the vehicle as if the vehicle was viewed from a bird's-eye perspective.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, further comprising:
estimating a width of the vehicle from the transformed 3-D bounding box of the vehicle; and
providing the estimated width to an autonomous control system.

### Claim 13

A non-transitory computer-readable storage medium storing computer program instructions executable to perform operations for training a machine-learned transformation model configured to receive an image of a vehicle from a reference viewpoint and output a target image of the vehicle from a target viewpoint different from the reference viewpoint, the operations comprising:
obtaining a training dataset of images that represent a set of computer simulated vehicles with a known set of characteristics, the training dataset including a training set of reference images that capture the computer simulated vehicles from one or more reference viewpoints and a corresponding training set of target images that capture the computer simulated vehicles from one or more target viewpoints different from the one or more reference viewpoints; and
training a set of parameters of the transformation model using the training dataset of images, the training comprising repeatedly performing iterations of:
generating a set of estimated target images by applying the transformation model with an estimated set of parameters to the training set of reference images,
determining a loss function indicating a difference between the set of estimated target images and the training set of target images, and
updating the set of parameters of the transformation model to reduce the loss function.

### Claim 14 (depends on 13)

The computer-readable storage medium of claim 13, wherein the set of computer simulated vehicles are computer-aided (CAD) generated models of vehicles, or computer simulated light detection and range (LIDAR) models of vehicles.

### Claim 15 (depends on 13)

The computer-readable storage medium of claim 13, the instructions further comprising:
obtaining a new image of a vehicle in a scene and a desired target viewpoint of the vehicle;
generating a predicted target image by applying the transformation model to a region of the new image containing the vehicle, the predicted target image predicting appearance of the vehicle from the desired target viewpoint; and
determining whether the vehicle is present in the region of the new image using the predicted target image.

### Claim 16 (depends on 13)

The computer-readable storage medium of claim 13, wherein the transformation model is further configured to a label indicating a desired target viewpoint, wherein the training dataset further includes a training set of labels indicating the one or more target viewpoints of the training set of target images, and wherein generating the set of estimated target images further comprises applying the transformation model with the estimated set of parameters to the training set of labels.

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein a label in the training set of labels is represented as a set of coordinates or a set of categories of viewpoints.

### Claim 18 (depends on 13)

The computer-readable storage medium of claim 13, wherein the one or more reference viewpoints are viewpoints showing side parts of the computer simulated vehicles.

### Claim 19

A non-transitory computer-readable storage medium storing computer program instructions executable to perform operations for training an object detection model configured to receive an image of a vehicle and output a set of 3-D bounding box coordinates of the vehicle in the image, comprising:
obtaining a training dataset of images that represent a set of computer simulated vehicles with a known set of characteristics, the characteristics including geometric information of the vehicles in the images;
obtaining a training set of coordinates for the images that represent 3-D bounding boxes of the computer simulated vehicles, the 3-D bounding boxes enclosing an outer boundary of the computer simulated vehicles, and the training set of coordinates for the 3-D bounding boxes determined based on the geometric information of the computer simulated vehicles; and
training a set of parameters of the object detection model using the training dataset of images, the training comprising repeatedly performing iterations of:
generating a set of estimated coordinates by applying the object detection model with an estimated set of parameters to the training dataset of images,
determining a loss function indicating a difference between the set of estimated coordinates and the training set of coordinates for the computer simulated vehicles, and
updating the set of parameters of the object detection model to reduce the loss function.

### Claim 20 (depends on 19)

The computer-readable storage medium of claim 19, wherein the set of computer simulated vehicles are computer-aided (CAD) generated models of vehicles, or computer simulated light detection and ranging (LIDAR) models of vehicles.

### Claim 21 (depends on 19)

The computer-readable storage medium of claim 19, the instructions further comprising:
obtaining a training set of orientations for the computer simulated vehicles that represent displacement of the computer simulated vehicles with respect to a reference axis,
wherein training the set of parameters further comprises:
generating a set of estimated orientations by applying the object detection model with the estimated set of parameters to the training dataset of images, and
wherein the loss function further indicates a difference between the set of estimated orientations and the training set of orientations for the computer simulated vehicles.

### Claim 22 (depends on 19)

The computer-readable storage medium of claim 19, the instructions further comprising:
obtaining a training set of categories for the computer simulated vehicles that represent classifications of the computer simulated vehicles with respect to their structure or functions,
wherein training the set of parameters further comprises:
generating a set of estimated categories by applying the object detection model with the estimated set of parameters to the training dataset of images, and
wherein the loss function further indicates a difference between the set of estimated categories and the training set of categories for the computer simulated vehicles.

### Claim 23 (depends on 19)

The computer-readable storage medium of claim 19, the instructions further comprising:
obtaining a new image of a vehicle in a scene;
generating a predicted set of coordinates of the vehicle by applying the object detection model to the new image, the predicted set of coordinates representing an estimated 3-D bounding box enclosing the vehicle; and
generating a transformed set of coordinates from the predicted set of coordinates, the transformed set of coordinates representing a transformed 3-D bounding box of the vehicle as if the vehicle was viewed from a bird's-eye perspective.

### Claim 24 (depends on 23)

The computer-readable storage medium of claim 23, the instructions further comprising:
estimating a width of the vehicle from the transformed 3-D bounding box of the vehicle; and
providing the estimated width to an autonomous control system.

### Claim 25

An autonomous control vehicle including a machine-learned transformation model stored on a computer-readable medium manufactured by the process of:
obtaining a training dataset of images that represent a set of computer simulated vehicles with a known set of characteristics, the training dataset including a training set of reference images that capture the computer simulated vehicles from one or more reference viewpoints and a corresponding training set of target images that capture the computer simulated vehicles from one or more target viewpoints different from the one or more reference viewpoints; and
training a set of parameters of the transformation model using the training dataset of images, the training comprising repeatedly performing iterations of:
generating a set of estimated target images by applying the transformation model with an estimated set of parameters to the training set of reference images,
determining a loss function indicating a difference between the set of estimated target images and the training set of target images, and
updating the set of parameters of the transformation model to reduce the loss function.

### Claim 26

An autonomous control vehicle including a machine-learned a machine-learned object detection model stored on a computer-readable medium manufactured by the process of:
obtaining a training dataset of images that represent a set of computer simulated vehicles with a known set of characteristics, the characteristics including geometric information of the vehicles in the images;
obtaining a training set of coordinates for the images that represent 3-D bounding boxes of the computer simulated vehicles, the 3-D bounding boxes enclosing an outer boundary of the computer simulated vehicles, and the training set of coordinates for the 3-D bounding boxes determined based on the geometric information of the computer simulated vehicles; and
training a set of parameters of the object detection model using the training dataset of images, the training comprising repeatedly performing iterations of:
generating a set of estimated coordinates by applying the object detection model with an estimated set of parameters to the training dataset of images,
determining a loss function indicating a difference between the set of estimated coordinates and the training set of coordinates for the computer simulated vehicles, and
updating the set of parameters of the object detection model to reduce the loss function.