# US 11,113,586 B2 · Method and electronic device for retrieving an image and computer readable storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,586 B2
- Document ID: US11113586B2
- Status: Granted
- Kind: B2
- Application number: 16/531,644
- Filed: 2019-08-05
- Published: 2020-07-30
- Granted: 2021-09-07
- Earliest claimed: 2019-01-29
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: Boe Technology Group Co., Ltd.
- Inventors: Tingting Wang
- Web page: https://patentgrep.com/patent/US11113586B2

## CPC

- G06N 3/08: Learning methods
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/253: {of extracted features}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

According to the embodiments of the present application, there are proposed a method and electronic device for retrieving an image, and computer readable storage medium. The method includes: processing an image to be retrieved using a first neural network to determine a local feature vector of the image to be retrieved; processing the image to be retrieved using a second neural network to determine a global feature vector of the image to be retrieved; and determining, based on the local feature vector and the global feature vector, an image having a similarity to the image to be retrieved which is higher than a similarity threshold.

## Claims

### Claim 1

A method for retrieving an image, comprising:
processing an image to be retrieved using a first neural network to determine a local feature vector of the image to be retrieved;
processing the image to be retrieved using a second neural network to determine a global feature vector of the image to be retrieved; and
determining, based on the local feature vector and the global feature vector, an image having a similarity to the image to be retrieved which is higher than a similarity threshold;
wherein the first neural network is trained by using a loss function as follows:
Lt(ya,yp,yn)=max(∥ya−yp∥22−∥ya−yn∥22+α,0),
where Lt represents a loss function for the first neural network, ya is a feature vector of a standard image, yp is a feature vector of a positive sample, yn is a feature vector of a negative sample, ∥⋅∥22 represents a square of 2-norm of a vector, max( ) represents a maximum value solving function, and α is margin value.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the first neural network is trained using a plurality of training image data having different resolutions of a training image, and the first neural network is used for processing a plurality of image data to be retrieved having different resolutions of the image to be retrieved.

### Claim 3 (depends on 2)

The method according to claim 2, wherein a number of pixels of the shortest side of the plurality of training image data having different resolutions or the plurality of training image data having different resolutions comprises at least two of 256, 384, 512, 640, and 768.

### Claim 4 (depends on 2)

The method according to claim 2, wherein the first neural network comprises the following convolutional layers:
a first convolutional layer having 96 convolution kernels with a dimension of 11*11*3;
a second convolutional layer having 256 convolution kernels with a dimension of 5*5*96;
a third convolutional layer having 384 convolution kernels with a dimension of 3*3*256;
a fourth convolutional layer having 384 convolution kernels with a dimension of 3*3*384;
a fifth convolutional layer having 256 convolution kernels with a dimension of 3*3*384;
a sixth convolutional layer having 4096 convolution kernels with a dimension of 1*1*256; and
a seventh convolutional layer having 4096 convolution kernels with a dimension of 13*13*4096.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the first neural network further comprises a max pooling layer and a sum pooling layer subsequent to the seventh convolutional layer.

### Claim 6 (depends on 1)

The method according to claim 1, wherein α is defined as:
α=0.5*∥yp−yn∥22

### Claim 7 (depends on 4)

The method according to claim 4, wherein the step of processing an image to be retrieved using a first neural network to determine a local feature vector of the image to be retrieved comprises:
processing, by using each convolutional layer in the first neural network, a plurality of image data to be retrieved having different resolutions of the image to be retrieved, and determining a plurality of receptive fields respectively having a maximum activation value in a plurality of feature maps for the respective resolutions as an output; and
processing the plurality of receptive fields using a sum pooling layer in the first neural network to determine the local feature vector.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the second neural network comprises the following convolutional layers:
a first convolutional layer having 96 convolution kernels with a dimension of 11*11*3;
a second convolutional layer having 256 convolution kernels with a dimension of 5*5*96;
a third convolutional layer having 384 convolution kernels with a dimension of 3*3*256;
a fourth convolutional layer having 384 convolution kernels with a dimension of 3*3*384;
a fifth convolutional layer having 256 convolution kernels with a dimension of 3*3*384;
a first fully connected layer with a dimension of 1*4096; and
a second fully connected layer with a dimension of 1*4096.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the second neural network further has a spatial transformer network between the fifth convolutional layer and the first fully connected layer.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the loss function used for training the second neural network is a loss function as follows:
L
s
⁡
(
y
⁢
1
,
⁢
y
⁢
2
,
⁢
y
)
=
(
1
-
y
)
2
⁢

y
⁢
⁢
1
-
y
⁢
⁢
2

2
2
+
y
2
⁢
max
⁡
(
m
-

y
⁢
⁢
1
-
y
⁢
⁢
2

2
2
,
0
)
,
where Ls represents a loss function for the second neural network, y1 and y2 are feature vectors of two sample images respectively, y is a correct label indicating whether the two sample images are similar, ∥⋅∥22 represents a square of 2-norm of a vector, max( ) represents a maximum value solving function, and m is a margin value.

### Claim 11 (depends on 2)

The method according to claim 2, wherein the loss function used for training the first neural network and the second neural network at the same time is a loss function as follows:
L
=
L
t
+
λ
⁢
L
s
L
t
⁡
(
y
⁢
a
,
y
⁢
p
,
y
⁢
n
)
=
max
⁡
(

ya
-
yp

2
2
-

ya
-
yn

2
2
+
α
,
⁢
0
)
L
s
⁡
(
y
⁢
1
,
⁢
y
⁢
2
,
y
)
=
(
1
-
y
)
2
⁢

y
⁢
⁢
1
-
y
⁢
⁢
2

2
2
+
y
2
⁢
max
⁡
(
m
-

y
⁢
⁢
1
-
y
⁢
⁢
2

2
2
,
0
)
where L is a total loss function, Lt represents a loss function for the first neural network, and Ls represents a loss function for the second neural network, and
where ya is a feature vector of a standard image, yp is a feature vector of a positive sample, yn is a feature vector of a negative sample, ∥⋅∥22 represents a square of 2-norm of a vector, max( ) represents a maximum value solving function, α is a margin value, y1 and y2 are feature vectors of two sample images respectively, y is a correct label indicating whether two input images are similar, ∥⋅∥22 represents a square of 2-norm of a vector, and m is a margin value.

### Claim 12

An electronic device for retrieving an image, comprising:
a processor; and
a memory having stored thereon instructions which, when executed by the processor, cause the processor to perform the following steps:
processing an image to be retrieved using a first neural network to determine a local feature vector of the image to be retrieved;
processing the image to be retrieved using a second neural network to determine a global feature vector of the image to be retrieved; and
determining, based on the local feature vector and the global feature vector, an image having a similarity to the image to be retrieved which is higher than a similarity threshold; and
the method further comprises training the first neural network by using a loss function as follows:
Lt(ya,yp,yn)=max(∥ya−yp∥22−∥ya−yn∥22+α,0),
where Lt represents a loss function for the first neural network, ya is a feature vector of a standard image, yp is a feature vector of a positive sample, yn is a feature vector of a negative sample, ∥⋅∥22 represents a square of 2-norm of a vector, max( ) represents a maximum value solving function, and α is margin value.

### Claim 13 (depends on 12)

The electronic device according to claim 12, wherein the first neural network is trained using a plurality of training image data having different resolutions of a training image, and the first neural network is used for processing a plurality of image data to be retrieved having different resolutions of the image to be retrieved.

### Claim 14 (depends on 12)

The electronic device according to claim 12, wherein the step of processing an image to be retrieved using a first neural network to determine a local feature vector of the image to be retrieved comprises:
processing, by using each convolutional layer in the first neural network, a plurality of image data to be retrieved having different resolutions of the image to be retrieved, and determining a plurality of receptive fields respectively having a maximum activation value in a plurality of feature maps for the respective resolutions as an output; and
processing the plurality of receptive fields using a sum pooling layer in the first neural network to determine the local feature vector.

### Claim 15 (depends on 12)

The electronic device according to claim 12, wherein the loss function used for training the second neural network is a loss function as follows:
L
s
⁡
(
y
⁢
1
,
⁢
y
⁢
2
,
⁢
y
)
=
(
1
-
y
)
2
⁢

y
⁢
⁢
1
-
y
⁢
⁢
2

2
2
+
y
2
⁢
max
⁡
(
m
-

y
⁢
⁢
1
-
y
⁢
⁢
2

2
2
,
0
)
,
where Ls represents a loss function for the second neural network, y1 and y2 are feature vectors of two sample images respectively, y is a correct label indicating whether the two sample images are similar, ∥⋅∥22 represents a square of 2-norm of a vector, max( ) represents a maximum value solving function, and in is a margin value.

### Claim 16

A non-transitory computer readable storage medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to perform the following steps:
processing an image to be retrieved using a first neural network to determine a local feature vector of the image to be retrieved;
processing the image to be retrieved using a second neural network to determine a global feature vector of the image to be retrieved; and
determining, based on the local feature vector and the global feature vector, an image having a similarity to the image to be retrieved which is higher than a similarity threshold; and
training the first neural network by using a loss function as follows:
Lt(ya,yp,yn)=max(∥ya−yp∥22−∥ya−yn∥22+α,0),
where Lt represents a loss function for the first neural network, ta is a feature vector of a standard image, yp is a feature vector of a positive sample, yn is a feature vector of a negative sample, ∥⋅∥22 represents a square of 2-norm of a vector, max( ) represents a maximum value solving function, and α is margin value.

### Claim 17 (depends on 16)

The non-transitory computer readable storage medium according to claim 16, wherein the first neural network is trained using a plurality of training image data having different resolutions of a training image, and the first neural network is used for processing a plurality of image data to be retrieved having different resolutions of the image to be retrieved.

### Claim 18 (depends on 16)

The non-transitory computer readable storage medium according to claim 16, wherein the step of processing an image to be retrieved using a first neural network to determine a local feature vector of the image to be retrieved comprises:
processing, by using each convolutional layer in the first neural network, a plurality of image data to be retrieved having different resolutions of the image to be retrieved, and determining a plurality of receptive fields respectively having a maximum activation value in a plurality of feature maps for the respective resolutions as an output; and
processing the plurality of receptive fields using a sum pooling layer in the first neural network to determine the local feature vector.