US 11,537,849 B2Granted

Computer-implemented method of training convolutional neural network, convolutional neural network, computer-implemented method using convolutional neural network, apparatus for training convolutional neural network, and computer-program product

AssigneeBoe Technology Group Co., Ltd.

InventorsDan Zhu, Lijie Zhang, Pablo Navarre Michelini, Hanwen Liu

Life of this patent

Estimating the term…

Loading patent…

Abstract

A computer-implemented method of training a convolutional neural network configured to morph content features of an input image with style features of a style image is provided. The computer-implemented method includes selecting a training style image; extracting style features of the training style image; selecting a training content image; extracting content features of the training content image; processing the training content image through the convolutional neural network to generate a training output image including the content features of the training content image morphed with the style features of the training style image; extracting content features and style features of the training output image; computing a total loss; and tuning the convolutional neural network based on the total loss including a content loss, a style loss, and a regularization loss.

Claims17

Dependent claims

A computer-implemented method of training a convolutional neural network configured to morph content features of an input image with style features of a style image, comprising:

selecting a training style image;
extracting style features of the training style image;
selecting a training content image;
extracting content features of the training content image;
processing the training content image through the convolutional neural network to generate a training output image comprising the content features of the training content image morphed with the style features of the training style image;
extracting content features and style features of the training output image;
computing a total loss;
and
tuning the convolutional neural network based on the total loss comprising a content loss, a style loss, and a regularization loss;
wherein tuning the convolutional neural network comprises tuning the convolutional neural network in part based on a regularization loss function generated using (1) a first equation Lm=ΣC=13VC[O]TLVC[O], and (2) a first matrix obtained using a second equation
L
(
i
,
j
)
=
Σ
k
❘
(
i
,
j
)
∈
W
k
(
δ
ij
-
1

W
k

⁢
(
1
+
1
ɛ

W
k

+
σ
k
2
⁢
(
I
i
-
μ
k
)
⁢
(
I
j
-
μ
k
)
T
)
)
;
wherein a matrix dimension of the first matrix is N×N, N is a total number of pixels of the training content image;
wherein L represents the first matrix;
L(i,j) represents a value of an element in an i-th row and a j-th column of the first matrix;
k stands for a k-th pixel;
Wk represents a window of pixels centered on the k-th pixel;
δij represents a Kronecker delta function;
σk2 represents a variance of values of elements in the window of pixels Wk;
|Wk| represents is a total number of pixels in the window of pixels Wk;
ε is a selected value;
Ii is a row vector representing an i-th row of a pixel matrix of the training content image;
Ij is a row vector representing an j-th row of the pixel matrix of the training content image;
μk is a row vector having a vector dimension same as vector dimensions of the row vector Ii and the row vector Ij;
and each element of μk is an average value of elements in the window of pixels Wk;
wherein Lm represents the regularization loss function;
C represents a sequential number of channels of the training content image;
O is a pixel matrix representing the training output image;
VC[O] represent a vectorized version of the training output image O in a C-th channel.

85 more elements

An apparatus for training a convolutional neural network configured to morph content features of an input image with style features of a style image, comprising:

a memory;
one or more processors;
and
wherein the memory and the one or more processors are connected with each other;
and
the memory stores computer-executable instructions for controlling the one or more processors to:
select a training style image;
extract style features of the training style image;
select a training content image;
extract content features of the training content image;
process the training content image through the convolutional neural network to generate a training output image comprising the content features of the training content image morphed with the style features of the training style image;
extract content features and style features of the training output image;
compute a total loss;
and
tune the convolutional neural network based on the total loss comprising a content loss, a style loss, and a regularization loss;
wherein tuning the convolutional neural network comprises tuning the convolutional neural network in part based on a regularization loss function generated using (1) a first equation Lm=ΣC=13VC[O]T LVC[O], and (2) a first matrix obtained using a second equation
L
(
i
,
j
)
=
Σ
k
❘
(
i
,
j
)
∈
W
k
(
δ
ij
-
1

W
k

⁢
(
1
+
1
ɛ

W
k

+
σ
k
2
⁢
(
I
i
-
μ
k
)
⁢
(
I
j
-
μ
k
)
T
)
)
;
wherein a matrix dimension of the first matrix is N×N, N is a total number of pixels of the training content image;
wherein L represents the first matrix;
L(i,j) represents a value of an element in an i-th row and a j-th column of the first matrix;
k stands for a k-th pixel;
Wk represents a window of pixels centered on the k-th pixel;
δij represents a Kronecker delta function;
σk2 represents a variance of values of elements in the window of pixels Wk;
|Wk| represents is a total number of pixels in the window of pixels Wk;
ε is a selected value;
Ii is a row vector representing an i-th row of a pixel matrix of the training content image;
Ij is a row vector representing an j-th row of the pixel matrix of the training content image;
μk is a row vector having a vector dimension same as vector dimensions of the row vector Ii and the row vector Ij;
and each element of μk is an average value of elements in the window of pixels Wk;
wherein Lm represents the regularization loss function;
C represents a sequential number of channels of the training content image;
O is a pixel matrix representing the training output image;
VC[O] represent a vectorized version of the training output image O in a C-th channel.

91 more elements

A computer-program product, for training a convolutional neural network configured to morph content features of an input image with style features of a style image, comprising a non-transitory tangible computer-readable medium having computer-readable instructions thereon, the computer-readable instructions being executable by a processor to cause the processor to perform:

selecting a training style image;
extracting style features of the training style image;
selecting a training content image;
extracting content features of the training content image;
processing the training content image through the convolutional neural network to generate a training output image comprising the content features of the training content image morphed with the style features of the training style image;
extracting content features and style features of the training output image;
computing a total loss;
and
tuning the convolutional neural network based on the total loss comprising a content loss, a style loss, and a regularization loss;
wherein tuning the convolutional neural network comprises tuning the convolutional neural network in part based on a regularization loss function generated using (1) a first equation Lm=ΣC=13VC[O]T LVC[O], and (2) a first matrix obtained using a second equation
L
(
i
,
j
)
=
∑
k
❘
(
i
,
j
)
∈
W
k
(
δ
i
⁢
j
-
1
❘
"\[LeftBracketingBar]"
W
k
❘
"\[RightBracketingBar]"
⁢
(
1
+
1
ε
❘
"\[LeftBracketingBar]"
W
k
❘
"\[RightBracketingBar]"
+
σ
k
2
⁢
(
I
i
-
μ
k
)
⁢
(
I
j
-
μ
k
)
T
)
)
;
wherein a matrix dimension of the first matrix is N×N, N is a total number of pixels of the training content image;
wherein L represents the first matrix;
L(i,j) represents a value of an element in an i-th row and a j-th column of the first matrix;
k stands for a k-th pixel;
Wk represents a window of pixels centered on the k-th pixel;
δij represents a Kronecker delta function;
σk2 represents a variance of values of elements in the window of pixels Wk;
|Wk| represents is a total number of pixels in the window of pixels Wk;
ε is a selected value;
Ii is a row vector representing an i-th row of a pixel matrix of the training content image;
Ij is a row vector representing an j-th row of the pixel matrix of the training content image;
μk is a row vector having a vector dimension same as vector dimensions of the row vector Ii and the row vector Ij;
and each element of μk is an average value of elements in the window of pixels Wk;
wherein Lm represents the regularization loss function;
C represents a sequential number of channels of the training content image;
O is a pixel matrix representing the training output image;
VC[O] represent a vectorized version of the training output image O in a C-th channel.

91 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/626,302Filed 22 Jul 2019
Priority: CN 201910005026.83 Jan 2019
Published: US 2021/0012181 A114 Jan 2021
Granted: 27 Dec 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification18 codes

G06N 3/0464
Architecture, e.g. interconnection topology: convolutional networks [CNN, ConvNet]
G06N 3/08
Neural networks: learning methods
G06F 17/16
Complex mathematical operations: matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
G06N 3/045
Architecture, e.g. interconnection topology: combinations of networks