# US 10,311,336 B1 · Method and device of neural network operations using a grid generator for converting modes according to classes of areas to satisfy level 4 of autonomous vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,336 B1
- Document ID: US10311336B1
- Status: Granted
- Kind: B1
- Application number: 16/254,525
- Filed: 2019-01-22
- Granted: 2019-06-04
- Earliest claimed: 2019-01-22
- Assignees: StradVision, Inc.
- Applicants: Stradvision, Inc.
- Inventors: Kye-Hyeon Kim; Yongjoong Kim; Insu Kim; Hak-Kyoung Kim; Woonhyun Nam; SukHoon Boo; Myungchul Sung; Donghun Yeo; Wooju Ryu; Taewoong Jang; Kyungjoong Jeong; Hongmo Je; Hojin Cho
- Web page: https://patentgrep.com/patent/US10311336B1

## CPC

- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2163: {Partitioning the feature space}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]}
- G06F 18/2431: Multiple classes
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method of neural network operations by using a grid generator is provided for converting modes according to classes of areas to satisfy level 4 of autonomous vehicles. The method includes steps of: (a) a computing device instructing a pair detector to acquire information on locations and classes of pairs for testing by detecting the pairs for testing; (b) the computing device instructing the grid generator to generate section information by referring to the information on the locations of the pairs for testing; (c) the computing device instructing a neural network to determine parameters for testing by referring to parameters for training which have been learned by using information on pairs for training; and (d) the computing device instructing the neural network to apply the neural network operations to a test image by using each of the parameters for testing to thereby generate one or more neural network outputs.

## Claims

### Claim 1

A method for neural network operations by using a grid generator, comprising steps of:
(a) a computing device, if a test image is acquired, instructing a pair detector to acquire information on locations and classes of one or more pairs for testing by detecting the pairs for testing, including objects for testing and non-objects for testing corresponding to the objects for testing, on the test image;
(b) the computing device instructing the grid generator to generate section information, which includes information on a plurality of subsections in the test image, by referring to the information on the locations of the pairs for testing;
(c) the computing device instructing a neural network to determine parameters for testing, to be used for applying the neural network operations to each of at least part of the subsections including each of the pairs for testing, by referring to parameters for training which have been learned by using information on pairs for training whose corresponding information on the classes is same as or similar to that on the pairs for testing; and
(d) the computing device instructing the neural network to apply the neural network operations to the test image by using each of the parameters for testing, corresponding to each of said at least part of the subsections, to thereby generate one or more neural network outputs.

### Claim 2 (depends on 1)

The method of claim 1, wherein, at the step of (b), the grid generator divides the test image by using a dynamic template, and wherein the dynamic template is provided by adjusting one or more boundaries included therein, such that (i) at least one of the pairs for testing is included in each of said at least part of the subsections of the dynamic template.

### Claim 3 (depends on 2)

The method of claim 2, wherein the grid generator manages information on the dynamic template, and wherein the dynamic template includes at least part of (i) at least one first boundary in a first direction and (ii) at least one second boundary in a second direction, some of which are adjustable.

### Claim 4 (depends on 2)

The method of claim 2, wherein the dynamic template includes row groups and one or more column groups, wherein the row groups have an upper row group, a middle row group including at least part of the column groups, and a lower row group, and wherein (i) at least one of the pairs for testing is included in each of said at least part of the subsections formed by the row groups and the column groups.

### Claim 5 (depends on 1)

The method of claim 1, before the step (a), comprising a step of:
(a0) the computing device instructing the neural network to learn the parameters for training by using information on each of the pairs for training included in one or more training images.

### Claim 6 (depends on 5)

The method of claim 5, wherein, at the step of (a0), the neural network learns the parameters for training by using information on each of the pairs for training included in each of at least part of partial areas, from a first partial area to a k-th partial area, included in the training images, and store information on classes of the pairs for training coupled with their corresponding parameters for training.

### Claim 7 (depends on 6)

The method of claim 6, wherein, on condition that there are first location information to k-th location information corresponding to the first partial area to the k-th partial area in each of the training images, the training images, in which respective relative variations of at least part of the first location information to the k-th location information are equal to or smaller than a predetermined threshold value, are selected.

### Claim 8 (depends on 6)

The method of claim 6, wherein, at the step of (c), (i) on condition that area of a specific object for testing is larger than that of a specific non-object for testing, which is included in a specific subsection along with the specific object for testing, the computing device determines a specific pair for testing, including the specific non-object for testing and the specific object for testing, as having first information on classes, and instructs the neural network to determine at least one specific parameter for testing among the parameters for testing, to be used for applying the neural network operations to the specific subsection, by referring to at least one first specific parameter for training among the parameters for training and (ii) on condition that the area of the specific object for testing is same as or smaller than that of the specific non-object, the computing device determines a specific pair for testing, including the specific non-object for testing and the specific object for testing, as having second class information on classes and instructs the neural network to determine at least one specific parameter for testing among the parameters for testing, to be used for applying the neural network operations to the specific subsection, by referring to at least one second specific parameter for training among the parameters for training.

### Claim 9 (depends on 1)

The method of claim 1, wherein, at the step of (d), the computing device instructs the neural network to generate the neural network outputs by applying one or more convolutional operations and one or more deconvolutional operations to the test image with the parameters for testing.

### Claim 10 (depends on 9)

The method of claim 9, wherein the test image represents road driving circumstances, the objects for testing represent objects existing on a road, and the neural network outputs are used for autonomous driving.

### Claim 11

A computing device for neural network operations by using a grid generator, comprising:
at least one memory that stores instructions; and
at least one processor configured to execute the instructions to: perform processes of (I) instructing a pair detector to acquire information on locations and classes of one or more pairs for testing by detecting the pairs for testing, including objects for testing and non-objects for testing corresponding to the objects for testing, on a test image, (II) instructing the grid generator to generate section information, which includes information on a plurality of subsections in the test image, by referring to the information on the locations of the pairs for testing, (III) instructing a neural network to determine parameters for testing, to be used for applying the neural network operations to each of at least part of the subsections including each of the pairs for testing, by referring to parameters for training which have been learned by using information on pairs for training whose corresponding information on the classes is same as or similar to that on the pairs for testing, and (IV) instructing the neural network to apply the neural network operations to the test image by using each of the parameters for testing, corresponding to each of said at least part of the subsections, to thereby generate one or more neural network outputs.

### Claim 12 (depends on 11)

The computing device of claim 11, wherein, at the process of (II), the grid generator divides the test image by using a dynamic template, and wherein the dynamic template is provided by adjusting one or more boundaries included therein, such that (i) at least one of the pairs for testing is included in each of said at least part of the subsections of the dynamic template.

### Claim 13 (depends on 12)

The computing device of claim 12, wherein the grid generator manages information on the dynamic template, and wherein the dynamic template includes at least part of (i) at least one first boundary in a first direction and (ii) at least one second boundary in a second direction, some of which are adjustable.

### Claim 14 (depends on 12)

The computing device of claim 12, wherein the dynamic template includes row groups and one or more column groups, wherein the row groups have an upper row group, a middle row group including at least part of the column groups, and a lower row group, and wherein (i) at least one of the pairs for testing is included in each of said at least part of the subsections formed by the row groups and the column groups.

### Claim 15 (depends on 11)

The computing device of claim 11, before the process (I), wherein the processor further performs a process of: (0) instructing the neural network to learn the parameters for training by using information on each of the pairs for training included in one or more training images.

### Claim 16 (depends on 15)

The computing device of claim 15, wherein, at the process of (0), the neural network learns the parameters for training by using information on each of the pairs for training included in each of at least part of partial areas, from a first partial area to a k-th partial area, included in the training images, and store information on classes of the pairs for training coupled with their corresponding parameters for training.

### Claim 17 (depends on 16)

The computing device of claim 16, wherein, on condition that there are first location information to k-th location information corresponding to the first partial area to the k-th partial area in each of the training images, the training images, in which respective relative variations of at least part of the first location information to the k-th location information are equal to or smaller than a predetermined threshold value, are selected.

### Claim 18 (depends on 16)

The computing device of claim 16, wherein, at the process of (III), (i) on condition that area of a specific object for testing is larger than that of a specific non-object for testing, which is included in a specific subsection along with the specific object for testing, the processor determines a specific pair for testing, including the specific non-object for testing and the specific object for testing, as having first information on classes, and instructs the neural network to determine at least one specific parameter for testing among the parameters for testing, to be used for applying the neural network operations to the specific subsection, by referring to at least one first specific parameter for training among the parameters for training and (ii) on condition that the area of the specific object for testing is same as or smaller than that of the specific non-object, the processor determines a specific pair for testing, including the specific non-object for testing and the specific object for testing, as having second class information on classes and instructs the neural network to determine at least one specific parameter for testing among the parameters for testing, to be used for applying the neural network operations to the specific subsection, by referring to at least one second specific parameter for training among the parameters for training.

### Claim 19 (depends on 11)

The computing device of claim 11, wherein, at the process of (IV), the processor instructs the neural network to generate the neural network outputs by applying one or more convolutional operations and one or more deconvolutional operations to the test image with the parameters for testing.

### Claim 20 (depends on 19)

The computing device of claim 19, wherein the test image represents road driving circumstances, the objects for testing represent objects existing on a road, and the neural network outputs are used for autonomous driving.