# US 10,311,337 B1 · Method and device for providing integrated feature map using ensemble of multiple outputs from convolutional neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,337 B1
- Document ID: US10311337B1
- Status: Granted
- Kind: B1
- Application number: 16/121,031
- Filed: 2018-09-04
- Granted: 2019-06-04
- Earliest claimed: 2018-09-04
- Assignees: StradVision, Inc.
- Applicants: Stradvision, Inc.
- Inventors: Kye-Hyeon Kim; Yongjoong Kim; Insu Kim; Hak-Kyoung Kim; Woonhyun Nam; SukHoon Boo; Myungchul Sung; Donghun Yeo; Wooju Ryu; Taewoong Jang; Kyungjoong Jeong; Hongmo Je; Hojin Cho
- Web page: https://patentgrep.com/patent/US10311337B1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24133: {Distances to prototypes}
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for providing an integrated feature map by using an ensemble of a plurality of outputs from a convolutional neural network (CNN) is provided. The method includes steps of: a CNN device (a) receiving an input image and applying a plurality of modification functions to the input image to thereby generate a plurality of modified input images; (b) applying convolution operations to each of the modified input images to thereby obtain each of modified feature maps corresponding to each of the modified input images; (c) applying each of reverse transform functions, corresponding to each of the modification functions, to each of the corresponding modified feature maps, to thereby generate each of reverse transform feature maps corresponding to each of the modified feature maps; and (d) integrating at least part of the reverse transform feature maps to thereby obtain an integrated feature map.

## Claims

### Claim 1

A method for providing an integrated feature map by using an ensemble of a plurality of outputs from a convolutional neural network (CNN) to isolate objects in an input image, comprising steps of:
(a) a CNN device receiving the input image and applying a plurality of modification functions to the input image to thereby generate a plurality of modified input images;
(b) the CNN device applying convolution operations to each of the modified input images to thereby obtain each of modified feature maps corresponding to each of the modified input images;
(c) the CNN device applying each of reverse transform functions, corresponding to each of the modification functions, to each of the corresponding modified feature maps, to thereby generate each of reverse transform feature maps corresponding to each of the modified feature maps; and
(d) the CNN device integrating at least part of the reverse transform feature maps to thereby obtain an integrated feature map, and assigning a label, corresponding to a highest channel among channel values per pixel in the integrated feature map, to its corresponding pixel, to thereby obtain a segmentation output image to isolate the objects in the input image.

### Claim 2 (depends on 1)

The method of claim 1, wherein a modification function Ti in the modification functions is determined as a function that modifies an input image I into n modified input images Ti(I),
wherein i is a natural number from 1 to n, and
wherein the modification function Ti is determined as a function having its corresponding reverse transform function T−1i(I))=1.

### Claim 3 (depends on 2)

The method of claim 2, wherein the modification function Ti modifies the input image I by using at least one of affine transform algorithm and thin-plate spline algorithm.

### Claim 4 (depends on 2)

The method of claim 2, wherein the respective modification functions use the same modification algorithm which has different detailed parameters.

### Claim 5 (depends on 1)

The method of claim 1, wherein the plurality of reverse transform feature maps have each segmentation score for each pixel, and
wherein, at the step of (d), the CNN device sums each of segmentation scores of each of pixels of each of the reverse transform feature maps by referring to each of relative locations of each of the pixels of the reverse transform feature maps, to thereby obtain the integrated feature map.

### Claim 6 (depends on 5)

The method of claim 5, wherein, at the step of (c), the CNN device moves each pixel of the modified feature maps, generated as a result of segmentation of the modified input images, to its corresponding location on a result of segmentation of the input image by using the reverse transform function, to thereby generate the plurality of reverse transform feature maps.

### Claim 7 (depends on 1)

The method of claim 1, further comprising a step of:
(e) the CNN device (i) calculating a loss based on a difference between an output value, obtained by referring to the integrated feature map, and a ground truth (GT) value and (ii) performing backpropagation in order to minimize the loss, to thereby optimize one or more parameters of the CNN device.

### Claim 8 (depends on 1)

The method of claim 1, wherein objects in the image include at least one of sky, a road, a building, an automobile, a background, and a person.

### Claim 9

A method for convolutional neural network (CNN)-testing a test image as an input image to isolate objects in the input image, comprising steps of:
(a) a test device, on condition that (i) a CNN device has received the input image for training and has applied a plurality of modification functions to the input image for training to thereby generate a plurality of modified input images for training; (ii) the CNN device has applied convolution operations to each of the modified input images for training to thereby obtain each of modified feature maps for training corresponding to each of the modified input images for training; (iii) the CNN device has applied each of reverse transform functions, corresponding to each of the modification functions, to each of the corresponding modified feature maps for training, to thereby generate each of reverse transform feature maps for training corresponding to each of the modified feature maps for training; (iv) the CNN device has integrated at least part of the reverse transform feature maps for training to thereby obtain an integrated feature map, and assigned a label, corresponding to a highest channel among channel values per pixel in the integrated feature map, to its corresponding pixel, to thereby obtain a segmentation output image to isolate the objects in the input image; and (v) the CNN device has calculated a loss based on a difference between an output value, obtained by referring to the integrated feature map, and a ground truth (GT) value and has performed backpropagation in order to minimize the loss, to thereby optimize one or more parameters of the CNN device, acquiring an input image for testing, wherein the test device includes the optimized parameters of the CNN device;
(b) the test device applying a plurality of modification functions to the input image for testing to thereby generate a plurality of modified input images for testing;
(c) the test device applying convolution operations to each of the modified input images for testing to thereby obtain each of modified feature maps for testing corresponding to each of the modified input images for testing;
(d) the test device applying each of reverse transform functions, corresponding to each of the modification functions, to each of the corresponding modified feature maps for testing, to thereby generate each of reverse transform feature maps for testing corresponding to each of the modified feature maps for testing; and
(e) the test device integrating at least part of the reverse transform feature maps for testing to thereby obtain an integrated feature map for testing.

### Claim 10 (depends on 9)

The method of claim 9, wherein a modification function Ti in the modification functions is determined as a function that modifies an input image I into n modified input images Ti(I),
wherein i is a natural number from 1 to n, and
wherein the modification function Ti is determined as a function having its corresponding reverse transform function T−1i(Ti(I))=1.

### Claim 11 (depends on 10)

The method of claim 10, wherein the respective modification functions use the same modification algorithm which has different detailed parameters.

### Claim 12 (depends on 9)

The method of claim 9, wherein the reverse transform feature maps for training and the reverse transform feature maps for testing have each segmentation score for each pixel,
wherein, at the step of (iv), the CNN device sums each of segmentation scores of each of pixels of each of the reverse transform feature maps for training by referring to each of relative locations of each of the pixels of the reverse transform feature maps for training, to thereby obtain the integrated feature map for training, and
wherein, at the step of (e), the CNN device sums each of segmentation scores of each of pixels of each of the reverse transform feature maps for testing by referring to each of relative locations of each of the pixels of the reverse transform feature maps for testing, to thereby obtain the integrated feature map for testing.

### Claim 13 (depends on 9)

The method of claim 9, wherein objects in the image include at least one of sky, a road, a building, an automobile, a background, and a person.

### Claim 14

A CNN device for providing an integrated feature map by using an ensemble of a plurality of outputs from a convolutional neural network (CNN) to isolate objects in an input image, comprising:
a receiving circuit that receives the input image as a training image; and
a processor for performing processes of (1) applying a plurality of modification functions to the input image to thereby generate a plurality of modified input images; (2) applying convolution operations to each of the modified input images to thereby obtain each of modified feature maps corresponding to each of the modified input images; (3) applying each of reverse transform functions, corresponding to each of the modification functions, to each of the corresponding modified feature maps, to thereby generate each of reverse transform feature maps corresponding to each of the modified feature maps; and (4) integrating at least part of the reverse transform feature maps to thereby obtain an integrated feature map, and assigning a label, corresponding to a highest channel among channel values per pixel in the integrated feature map, to its corresponding pixel, to thereby obtain a segmentation output image to isolate the objects in the input image.

### Claim 15 (depends on 14)

The CNN device of claim 14, wherein the modification function Ti in the modification functions is determined as a function that modifies an input image I into n modified input images Ti(I),
wherein i is a natural number from 1 to n, and
wherein the modification function Ti is determined as a function having its corresponding reverse transform function T−1i(Ti(I))=1.

### Claim 16 (depends on 15)

The CNN device of claim 15, wherein the modification function Ti modifies the input image I by using at least one of affine transform algorithm and thin-plate spline algorithm.

### Claim 17 (depends on 15)

The CNN device of claim 15, wherein the respective modification functions use the same modification algorithm which has different detailed parameters.

### Claim 18 (depends on 14)

The CNN device of claim 14, wherein the plurality of reverse transform feature maps have each segmentation score for each pixel, and
wherein, at the process of (4), the processor sums each of segmentation scores of each of pixels of each of the reverse transform feature maps by referring to each of relative locations of each of the pixels of the reverse transform feature maps, to thereby obtain the integrated feature map.

### Claim 19 (depends on 18)

The CNN device of claim 18, wherein, at the process of (3), the processor moves each pixel of the modified feature maps, generated as a result of segmentation of the modified input images, to its corresponding location on a result of segmentation of the input image by using the reverse transform function, to thereby generate the plurality of reverse transform feature maps.

### Claim 20 (depends on 14)

The CNN device of claim 14, wherein the processor further performs a process of:
(5) (i) calculating a loss based on a difference between an output value, obtained by referring to the integrated feature map, and a ground truth (GT) value and (ii) performing backpropagation in order to minimize the loss, to thereby optimize one or more parameters of the CNN device.

### Claim 21 (depends on 14)

The CNN device of claim 14, wherein objects in the image include at least one of sky, a road, a building, an automobile, a background, and a person.

### Claim 22

A test device for convolutional neural network (CNN)-testing a test image as an input image to isolate objects in the input image, the test device comprising:
a receiving circuit that acquires the input image for testing on condition that (i) a CNN device has received the input image for training and has applied a plurality of modification functions to the input image for training to thereby generate a plurality of modified input images for training; (ii) the CNN device has applied convolution operations to each of the modified input images for training to thereby obtain each of modified feature maps for training corresponding to each of the modified input images for training; (iii) the CNN device has applied each of reverse transform functions, corresponding to each of the modification functions, to each of the corresponding modified feature maps for training, to thereby generate each of reverse transform feature maps for training corresponding to each of the modified feature maps for training; (iv) the CNN device has integrated at least part of the reverse transform feature maps for training to thereby obtain an integrated feature map, and assigned a label, corresponding to a highest channel among channel values per pixel in the integrated feature map, to its corresponding pixel, to thereby obtain a segmentation output image to isolate the objects in the input image; and (v) the CNN device has calculated a loss based on a difference between an output value, obtained by referring to the integrated feature map, and a ground truth (GT) value and has performed backpropagation in order to minimize the loss, to thereby optimize one or more parameters of the CNN device, wherein the test device includes the optimized parameters of the CNN device; and
a processor for performing processes of (1) applying a plurality of modification functions to the input image for testing to thereby generate a plurality of modified input images for testing; (2) applying convolution operations to each of the modified input images for testing to thereby obtain each of modified feature maps for testing corresponding to each of the modified input images for testing; (3) applying each of reverse transform functions, corresponding to each of the modification functions, to each of the corresponding modified feature maps for testing, to thereby generate each of reverse transform feature maps for testing corresponding to each of the modified feature maps for testing; and (4) integrating at least part of the reverse transform feature maps for testing to thereby obtain an integrated feature map for testing.

### Claim 23 (depends on 22)

The test device of claim 22, wherein a modification function Ti in the modification functions is determined as a function that modifies an input image I into n modified input images Ti(I),
wherein i is a natural number from 1 to n, and
wherein the modification function Ti is determined as a function having its corresponding reverse transform function T−1i(Ti(I))=1.

### Claim 24 (depends on 23)

The test device of claim 23, wherein the respective modification functions use the same modification algorithm which has different detailed parameters.

### Claim 25 (depends on 22)

The test device of claim 22, wherein the reverse transform feature maps for training and the reverse transform feature maps for testing have each segmentation score for each pixel,
wherein, at the process of (iv), the processor sums each of segmentation scores of each of pixels of each of the reverse transform feature maps for training by referring to each of relative locations of each of the pixels of the reverse transform feature maps for training, to thereby obtain the integrated feature map for training, and
wherein, at the process of (4), the processor sums each of segmentation scores of each of pixels of each of the reverse transform feature maps for testing by referring to each of relative locations of each of the pixels of the reverse transform feature maps for testing, to thereby obtain the integrated feature map for testing.

### Claim 26 (depends on 22)

The test device of claim 22, wherein objects in the image include at least one of sky, a road, a building, an automobile, a background, and a person.