# US 12,008,465 B1 · Dynamic generation of data sets for training machine-trained network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,465 B1
- Document ID: US12008465B1
- Status: Granted
- Kind: B1
- Application number: 15/870,070
- Filed: 2018-01-12
- Granted: 2024-06-11
- Earliest claimed: 2017-12-29
- Assignees: Perceive Corporation
- Applicants: Perceive Corporation
- Inventors: Andrew Mihal; Steven Teig
- Web page: https://patentgrep.com/patent/US12008465B1

## CPC

- G06T 7/85: {Stereo camera calibration}
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/55: from multiple images
- G06T 7/593: from stereo images
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G08G 1/16: Anti-collision systems (road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08)
- H04N 13/128: Adjusting depth or disparity
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- G06N 3/048: Activation functions
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30261: Obstacle
- H04N 13/246: Calibration of cameras
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Some embodiments of the invention provide a novel method for training a multi-layer node network. Some embodiments train the multi-layer network using a set of inputs generated with random misalignments incorporated into the training data set. In some embodiments, the training data set is a synthetically generated training set based on a three-dimensional ground truth model as it would be sensed by a sensor array from different positions and with different deviations from ideal alignment and placement. Some embodiments dynamically generate training data sets when a determination is made that more training is required. Training data sets, in some embodiments, are generated based on training data sets for which the multi-layer node network has produced bad results.

## Claims

### Claim 1

A method of dynamically generating training data sets for training a machine-trained (MT) network for modifying images captured by a device comprising a plurality of image capture sensors, the training data sets comprising input data sets and expected output data sets, the method comprising:
generating a plurality of training data sets for a plurality of different sensor misalignments for training the MT network, each training data set comprising an input data set corresponding to a different misalignment of the plurality of image capture sensors;
propagating each of the generated training data sets through the MT network to produce an output data set, each output data set used to train the network based on a loss function that comprises a measurement of a difference between the produced output data set and its expected output data set;
determining whether further training of the MT network is required based on the differences between the produced output data sets and their expected output data sets;
after determining that further training of the MT network is required, generating at least one additional training data set for at least one additional sensor misalignment not included in the plurality of sensor misalignments for further training of the MT network; and
propagating the at least one additional training data set through the MT network to produce another output data set to train the MT network.

### Claim 2 (depends on 1)

The method of claim 1, wherein the determining and generating are performed iteratively until at least one of a set of stopping conditions has been met.

### Claim 3 (depends on 2)

The method of claim 2, wherein the set of stopping conditions comprises a condition that a measurement of a difference between the produced output data sets and the expected output data sets for a set of training data sets is less than a particular value.

### Claim 4 (depends on 3)

The method of claim 3, wherein the set of stopping conditions further comprises a condition that a certain number of training data sets has been generated.

### Claim 5 (depends on 1)

The method of claim 1, wherein the plurality of training data sets are virtual training data sets.

### Claim 6 (depends on 1)

The method of claim 1, wherein the MT network is trained to aid in image processing for a particular image capture device array.

### Claim 7 (depends on 6)

The method of claim 6, wherein the particular image capture device array comprises a plurality of image capture devices.

### Claim 8 (depends on 1)

The method of claim 1, wherein the at least one additional training data set is generated based on a set of training data sets for which the measurement of the difference between the produced output data set and the expected output data set is large relative to other sets of training data sets.

### Claim 9 (depends on 8)

The method of claim 8, wherein the set of training data sets on which the at least one additional training data set is based represents a set of training data sets with a common feature.

### Claim 10 (depends on 9)

The method of claim 9, wherein each training data set in the set of training data sets comprises a set of image data and the common feature is at least one of (i) shadows, (ii) objects at a certain distance, (iii) combinations of objects at certain distances, and (iv) objects in certain areas of an image.

### Claim 11 (depends on 1)

The method of claim 1, wherein:
the input data set of a particular training data set comprises a set of computer-generated images representing a same computer-generated scene, each image corresponding to an image captured by an image capture device in a virtual image capture device array placed in the computer-generated scene, each image representing the same computer-generated scene at the same time; and
the expected output data set of the particular training data set comprises a set of computer-generated values based on a position of the virtual image capture device array and objects in the computer-generated scene.

### Claim 12 (depends on 11)

The method of claim 11, wherein the set of computer-generated values is a set of depth values generated by a ray tracing operation from a position associated with the virtual image capture device array to an object in the computer-generated scene.

### Claim 13

A non-transitory machine readable medium storing a program which when executed by at least one processing unit dynamically generates training data sets for training a machine-trained (MT) network, the training data sets comprising input data sets and expected output data sets, the program comprising sets of instructions for:
generating a first group of training data sets for training the MT network;
propagating each generated training data set in the first group of training data sets through the MT network to produce an output data set, each output data set used to train the MT network based on a loss function that comprises a measurement of a difference between the produced output data set and its expected output data set;
determining whether further training of the MT network is required based on the measurements of the differences between the produced output data sets and their expected output data sets, said determining comprising identifying a subset of input data sets for which the MT network produced worse results than for other subsets of input data sets;
after determining that further training of the MT network is required, generating an additional group of training data sets related to the identified subset of input data sets for further training of the MT network; and
propagating the training data sets in the additional group through the MT network to train the MT network in order to address deficiency in the training of the MT network based on the identified subset of input data sets,
wherein the sets of instructions for determining, generating, and propagating are executed iteratively until at least one of a set of stopping conditions has been met.

### Claim 14 (depends on 13)

The non-transitory machine readable medium of claim 13, wherein the set of stopping conditions comprises a condition that the measurement of the differences between the produced output data sets and the expected output data sets for a set of training data sets is less than a particular value.

### Claim 15 (depends on 14)

The non-transitory machine readable medium of claim 14, wherein the set of stopping conditions further comprises a condition that a certain number of training data sets has been generated.

### Claim 16 (depends on 13)

The non-transitory machine readable medium of claim 13, wherein the first group of training data sets and the additional groups of training data sets are virtual training data sets.

### Claim 17 (depends on 13)

The non-transitory machine readable medium of claim 13, wherein the MT network is trained to aid in image processing for a particular image capture device array.

### Claim 18 (depends on 17)

The non-transitory machine readable medium of claim 17, wherein the particular image capture device array comprises a plurality of image capture devices.

### Claim 19

A non-transitory machine readable medium storing a program which when executed by at least one processing unit dynamically generates training data sets for training a machine-trained (MT) network, the training data sets comprising input data sets and expected output data sets, the program comprising sets of instructions for:
generating a first group of training data sets for training the MT network;
propagating each generated training data set in the first group of training data sets through the MT network to produce an output data set, each output data set used to train the MT network based on a loss function that comprises a measurement of a difference between the produced output data set and its expected output data set;
determining whether further training of the MT network is required based on the measurements of the differences between the produced output data sets and their expected output data sets, said determining comprising identifying a subset of input data sets for which the MT network produced worse results than for other subsets of input data sets;
after determining that further training of the MT network is required, generating a second group of training data sets related to the identified subset of input data sets for further training of the MT network, wherein the second group of training data sets is generated based on a group of training data sets for which the measurements of the differences between the produced output data sets and their expected output data sets are large relative to other groups of training data sets; and
propagating the training data sets in the second group through the MT network to train the MT network in order to address deficiency in the training of the MT network based on the identified subset of input data sets.

### Claim 20 (depends on 19)

The non-transitory machine readable medium of claim 19, wherein the group of training data sets on which the second group of training data sets is based represents a group of training data sets with a common feature.

### Claim 21 (depends on 20)

The non-transitory machine readable medium of claim 20, wherein each training data set in the first and second groups of training data sets comprises a set of image data and the common feature is at least one of (i) shadows, (ii) objects at a certain distance, (iii) combinations of objects at certain distances, and (iv) objects in certain areas of an image.

### Claim 22

A non-transitory machine readable medium storing a program which when executed by at least one processing unit dynamically generates training data sets for training a machine-trained (MT) network, the training data sets comprising input data sets and expected output data sets, the program comprising sets of instructions for:
generating a first group of training data sets for training the MT network, wherein:
the input data set of a particular training data set comprises a set of computer-generated images representing a same computer-generated scene, each image corresponding to an image captured by an image capture device in a virtual image capture device array placed in the computer-generated scene, each image representing the same computer-generated scene at the same time; and
the expected output data set of the particular training data set comprises a set of computer-generated values based on a position of the virtual image capture device array and objects in the computer-generated scene;
propagating each generated training data set in the first group of training data sets through the MT network to produce an output data set, each output data set used to train the MT network based on a loss function that comprises a measurement of a difference between the produced output data set and its expected output data set;
determining whether further training of the MT network is required based on the measurements of the differences between the produced output data sets and their expected output data sets, said determining comprising identifying a subset of input data sets for which the MT network produced worse results than for other subsets of input data sets;
after determining that further training of the MT network is required, generating a second group of training data sets related to the identified subset of input data sets for further training of the MT network; and
propagating the training data sets in the second group through the MT network to train the MT network in order to address deficiency in the training of the MT network based on the identified subset of input data sets.

### Claim 23 (depends on 22)

The non-transitory machine readable medium of claim 22, wherein the set of computer-generated values is a set of depth values generated by a ray tracing operation from a position associated with the virtual image capture device array to an object in the computer-generated scene.