# US 10,373,317 B1 · Learning method and learning device for attention-driven image segmentation by using at least one adaptive loss weight map to be used for updating HD maps required to satisfy level 4 of autonomous vehicles and testing method and testing device using the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,317 B1
- Document ID: US10373317B1
- Status: Granted
- Kind: B1
- Application number: 16/254,545
- Filed: 2019-01-22
- Granted: 2019-08-06
- Earliest claimed: 2019-01-22
- Assignees: StradVision, Inc.
- Applicants: Stradvision, Inc.
- Inventors: Kye-Hyeon Kim; Yongjoong Kim; Insu Kim; Hak-Kyoung Kim; Woonhyun Nam; SukHoon Boo; Myungchul Sung; Donghun Yeo; Wooju Ryu; Taewoong Jang; Kyungjoong Jeong; Hongmo Je; Hojin Cho
- Web page: https://patentgrep.com/patent/US10373317B1

## CPC

- G06T 7/11: Region-based segmentation
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06F 18/2413: based on distances to training or reference patterns
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for an attention-driven image segmentation by using at least one adaptive loss weight map is provided to be used for updating HD maps required to satisfy level 4 of autonomous vehicles. By this method, vague objects such as lanes and road markers at distance may be detected more accurately. Also, this method can be usefully performed in military, where identification of friend or foe is important, by distinguishing aircraft marks or military uniforms at distance. The method includes steps of: a learning device instructing a softmax layer to generate softmax scores; instructing a loss weight layer to generate loss weight values by applying loss weight operations to predicted error values generated therefrom; and instructing a softmax loss layer to generate adjusted softmax loss values by referring to initial softmax loss values, generated by referring to the softmax scores and their corresponding GTs, and the loss weight values.

## Claims

### Claim 1

A method for image segmentation by using at least one adaptive loss weight map, comprising steps of:
(a) a learning device, when at least one input image is acquired, (i) instructing an encoding layer of a CNN to generate at least one feature map by applying one or more convolution operations to the input image, and (ii) instructing a decoding layer of the CNN to generate each of segmentation scores corresponding to each of pixels of the input image by applying one or more deconvolution operations to the feature map;
(b) the learning device instructing a softmax layer of the CNN to generate each of softmax scores corresponding to each of the pixels of the input image by applying softmax operations to each of the segmentation scores;
(c) the learning device instructing a loss weight layer of the CNN (i) to generate each of predicted error values corresponding to each of the pixels of the input image by referring to the softmax scores and their corresponding GT values, and (ii) to generate each of loss weight values corresponding to each of the pixels of the input image by applying loss weight operations to each of the predicted error values, wherein the loss weight operations allow differences among each of the loss weight values to be larger than those among each of the predicted error values; and
(d) the learning device (i) instructing a softmax loss layer of the CNN (1) to generate each of initial softmax loss values corresponding to each of the pixels of the input image by referring to each of the softmax scores and their corresponding GT values, and (2) to generate each of adjusted softmax loss values corresponding to each of the pixels of the input image by referring to each of the initial softmax loss values and each of the loss weight values, and (ii) learning parameters of the CNN by backpropagating the adjusted softmax loss values.

### Claim 2 (depends on 1)

The method of claim 1, wherein, at the step of (c), the loss weight operations follow a formula below,
loss weight value for i−th pixel=αi×Ei+βi
wherein αi is a scaling parameter corresponding to an i−th pixel among the pixels, Ei is the predicted error value corresponding to the i−th pixel among the pixels, βi is a bias parameter corresponding to an i−th pixel among the pixels, i is an integer from 1 to s, and s is the number of the pixels included in the input image.

### Claim 3 (depends on 1)

The method of claim 1, wherein, at the step of (d), the learning device instructs the softmax loss layer of the CNN to generate each of the adjusted softmax loss values by multiplying each of the loss weight values and its corresponding initial softmax loss value.

### Claim 4 (depends on 1)

The method of claim 1, wherein, at the step of (d), the initial softmax loss values are generated by operations following a formula below,
sum
⁢
⁢
of
⁢
⁢
initial
⁢
⁢
softmax
⁢
⁢
loss
=
-
1
S
⁢
∑
i
s
⁢
log
⁡
(
P
⁡
(
i
)
l
i
)
wherein, s is the number of the pixels included in the input image, li is an one-hot-encoding vector representing to which cluster an i−th pixel belongs on its corresponding GT value, and P(i) is a vector corresponding to a softmax score for the i−th pixel.

### Claim 5 (depends on 1)

The method of claim 1, wherein the input image includes information on a road driving circumstance, and
wherein the learning device learns the parameters of the CNN by backpropagating the adjusted softmax loss generated by referring to each of the loss weight values and its corresponding initial softmax loss value, on condition that the loss weight layer determines each of the loss weight values by assigning first loss weight values corresponding to at least one lane part of the input image and assigning second loss weight values corresponding to a background part of the input image, the first loss weight values being larger than the second loss weight values.

### Claim 6

A testing method for image segmentation by using at least one adaptive loss weight map, comprising steps of:
(a) on condition that (I) a learning device, when at least one training image has been acquired, (i) has instructed an encoding layer of a CNN to generate at least one feature map for training by applying one or more convolution operations to the training image, and (ii) has instructed a decoding layer of the CNN to generate each of segmentation scores for training corresponding to each of pixels of the training image by applying one or more deconvolution operations to the feature map for training; (II) the learning device has instructed a softmax layer of the CNN to generate each of softmax scores for training corresponding to each of the pixels of the training image by applying softmax operations to each of the segmentation scores for training; (III) the learning device has instructed a loss weight layer of the CNN (i) to generate each of predicted error values corresponding to each of the pixels of the training image by referring to the softmax scores for training and their corresponding GT values, and (ii) to generate each of loss weight values corresponding to each of the pixels of the training image by applying loss weight operations to each of the predicted error values, wherein the loss weight operations allow differences among each of the loss weight to be larger than those among each of the predicted error values; and (IV) the learning device (i) has instructed a softmax loss layer of the CNN (1) to generate each of initial softmax loss values corresponding to each of the pixels of the training image by referring to each of the softmax scores for training and their corresponding GT values, and (2) to generate each of adjusted softmax loss values corresponding to each of the pixels of the training image by referring to each of the initial softmax loss values and each of the loss weight values for training, and (ii) has learned parameters of the CNN by backpropagating the adjusted softmax loss values; when at least one test image is acquired, a testing device (i) instructing an encoding layer of a CNN to generate at least one feature map for testing by applying one or more convolution operations to the test image, and (ii) instructing a decoding layer of the CNN to generate each of segmentation scores for testing corresponding to each of pixels of the test image by applying one or more deconvolution operations to the feature map for testing; and
(b) the testing device instructing a softmax layer of the CNN to generate each of softmax scores for testing corresponding to each of the pixels of the test image by applying softmax operations to each of the segmentation scores for testing.

### Claim 7

A learning device for image segmentation by using at least one adaptive loss weight map, comprising:
at least one memory that stores instructions; and
at least one processor configured to execute the instructions to: perform processes of (I) (i) instructing an encoding layer of a CNN to generate at least one feature map by applying one or more convolution operations to the input image, and (ii) instructing a decoding layer of the CNN to generate each of segmentation scores corresponding to each of pixels of the input image by applying one or more deconvolution operations to the feature map, (II) instructing a softmax layer of the CNN to generate each of softmax scores corresponding to each of the pixels of the input image by applying softmax operations to each of the segmentation scores, (III) instructing a loss weight layer of the CNN (i) to generate each of predicted error values corresponding to each of the pixels of the input image by referring to the softmax scores and their corresponding GT values, and (ii) to generate each of loss weight values corresponding to each of the pixels of the input image by applying loss weight operations to each of the predicted error values, wherein the loss weight operations allow differences among each of the loss weight values to be larger than those among each of the predicted error values, and (IV) (i) instructing a softmax loss layer of the CNN (1) to generate each of initial softmax loss values corresponding to each of the pixels of the input image by referring to each of the softmax scores and their corresponding GT values, and (2) to generate each of adjusted softmax loss values corresponding to each of the pixels of the input image by referring to each of the initial softmax loss values and each of the loss weight values, and (ii) learning parameters of the CNN by backpropagating the adjusted softmax loss values.

### Claim 8 (depends on 7)

The learning device of claim 7, wherein, at the process of (III), the loss weight operations follow a formula below,
loss weight value for i−th pixel=αi×Ei+βi
wherein αi is a scaling parameter corresponding to an i-th pixel among the pixels, Ei is the predicted error value corresponding to the i−th pixel among the pixels, βi is a bias parameter corresponding to an i−th pixel among the pixels, i is an integer from 1 to s, and s is the number of the pixels included in the input image.

### Claim 9 (depends on 7)

The learning device of claim 7, wherein, at the process of (IV), the processor instructs the softmax loss layer of the CNN to generate each of the adjusted softmax loss values by multiplying each of the loss weight values and its corresponding initial softmax loss value.

### Claim 10 (depends on 7)

The learning device of claim 7, wherein, at the process of (IV), the initial softmax loss values are generated by operations following a formula below,
sum
⁢
⁢
of
⁢
⁢
initial
⁢
⁢
softmax
⁢
⁢
loss
=
-
1
S
⁢
∑
i
s
⁢
log
⁡
(
P
⁡
(
i
)
l
i
)
wherein, s is the number of the pixels included in the input image, li is an one-hot-encoding vector representing to which cluster an i−th pixel belongs on its corresponding GT value, and P(i) is a vector corresponding to a softmax score for the i−th pixel.

### Claim 11 (depends on 7)

The learning device of claim 7, wherein the input image includes information on a road driving circumstance, and
wherein the processor learns the parameters of the CNN by backpropagating the adjusted softmax loss generated by referring to each of the loss weight values and its corresponding initial softmax loss value, on condition that the loss weight layer determines each of the loss weight values by assigning first loss weight values corresponding to at least one lane part of the input image and assigning second loss weight values corresponding to a background part of the input image, the first loss weight values being larger than the second loss weight values.

### Claim 12

A testing device for image segmentation by using at least one adaptive loss weight map, comprising:
at least one memory that stores instructions; and
at least one processor, on condition that (I) a learning device, when at least one training image has been acquired, (i) has instructed an encoding layer of a CNN to generate at least one feature map for training by applying one or more convolution operations to the training image, and (ii) has instructed a decoding layer of the CNN to generate each of segmentation scores for training corresponding to each of pixels of the training image by applying one or more deconvolution operations to the feature map for training; (II) the learning device has instructed a softmax layer of the CNN to generate each of softmax scores for training corresponding to each of the pixels of the training image by applying softmax operations to each of the segmentation scores for training; (III) the learning device has instructed a loss weight layer of the CNN (i) to generate each of predicted error values corresponding to each of the pixels of the training image by referring to the softmax scores for training and their corresponding GT values, and (ii) to generate each of loss weight values corresponding to each of the pixels of the training image by applying loss weight operations to each of the predicted error values, wherein the loss weight operations allow differences among each of the loss weight to be larger than those among each of the predicted error values; and (IV) the learning device (i) has instructed a softmax loss layer of the CNN (1) to generate each of initial softmax loss values corresponding to each of the pixels of the training image by referring to each of the softmax scores for training and their corresponding GT values, and (2) to generate each of adjusted softmax loss values corresponding to each of the pixels of the training image by referring to each of the initial softmax loss values and each of the loss weight values for training, and (ii) has learned parameters of the CNN by backpropagating the adjusted softmax loss values; configured to execute the instructions to: perform processes of (I) (i) instructing an encoding layer of a CNN to generate at least one feature map for testing by applying one or more convolution operations to at least one test image, and (ii) instructing a decoding layer of the CNN to generate each of segmentation scores for testing corresponding to each of pixels of the test image by applying one or more deconvolution operations to the feature map for testing, and (II) instructing a softmax layer of the CNN to generate each of softmax scores for testing corresponding to each of the pixels of the test image by applying softmax operations to each of the segmentation scores for testing.