# US 11,308,598 B2 · Quality assessment of an image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,598 B2
- Document ID: US11308598B2
- Status: Granted
- Kind: B2
- Application number: 16/790,791
- Filed: 2020-02-14
- Published: 2020-06-11
- Granted: 2022-04-19
- Earliest claimed: 2019-02-14
- Assignees: Sharif University of Technology
- Applicants: Hatef Otroshi Shahreza; Arash Amini; Hamid Behroozi
- Inventors: Hatef Otroshi Shahreza; Arash Amini; Hamid Behroozi
- Web page: https://patentgrep.com/patent/US11308598B2

## CPC

- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30168: Image quality inspection
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for quality assessment of an image. The method includes generating an (i+1)th plurality of feature maps from an image, obtaining an (i+1)th feature set of a plurality of feature sets from the (i+1)th plurality of feature maps, and extracting a score distribution for a plurality of scores from the plurality of feature sets. The plurality of scores are associated with the image. The score distribution is extracted by feeding the plurality of feature sets to a first (1st) fully connected layer of a plurality of fully connected layers. The plurality of fully connected layers are associated with a convolutional neural network.

## Claims

### Claim 1

A method for quality assessment of an image, the method comprising:
generating, utilizing one or more processors, an (i+1)th plurality of feature maps from the image by:
feeding the image to a first convolutional layer of a plurality of convolutional layers associated with a convolutional neural network, the image comprising a first plurality of feature maps; and
extracting the (i+1)th plurality of feature maps by applying an ith plurality of feature maps to an ith convolutional layer of the plurality of convolutional layers, where 1≤i≤N and N is a number of the plurality of convolutional layers;
obtaining, utilizing the one or more processors, an (i+1)th feature set of a plurality of feature sets from the (i+1)th plurality of feature maps; and
extracting, utilizing the one or more processors, a score distribution for a plurality of scores associated with the image from the plurality of feature sets by:
generating a first fully connected output of a plurality of fully connected outputs by applying the plurality of feature sets to a first fully connected layer of a plurality of fully connected layers associated with the convolutional neural network; and
generating a kth fully connected output of the plurality of fully connected outputs by applying a (k−1)th fully connected output of a plurality of fully connected outputs to a kth fully connected layer of the plurality of fully connected layers,
where 1≤k≤L, L is a number of the plurality of fully connected layers, a 0th fully connected output comprises the plurality of feature sets, and an Lth fully connected output of the plurality of fully connected outputs comprises the score distribution.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the (i+1)th plurality of feature maps comprises training the convolutional neural network prior to feeding the image to the first convolutional layer, training the convolutional neural network comprising:
initializing the convolutional neural network by a plurality of initial weights; and
repeating an iterative process until a termination condition is satisfied, the iterative process comprising:
generating a plurality of updated weights by feeding a training image of a plurality of training images to the convolutional neural network; and
replacing the plurality of initial weights with the plurality of updated weights.

### Claim 3 (depends on 2)

The method of claim 2, wherein generating the plurality of updated weights comprises:
generating an (i+1)th plurality of training feature maps from the training image by:
feeding the training image to the first convolutional layer of the plurality of convolutional layers; and
extracting the (i+1)th plurality of training feature maps from an output of the ith convolutional layer;
obtaining an (i+1)th training feature set of a plurality of training feature sets from the (i+1)th plurality of training feature maps;
extracting a training distribution from the plurality of training feature sets by feeding the plurality of training feature sets to the first fully connected layer of the plurality of fully connected layers;
generating a plurality of adjustment values by minimizing a loss function, each of the plurality of adjustment values associated with a respective initial weight of the plurality of initial weights; and
obtaining the plurality of updated weights by adding each of the plurality of adjustment values to a respective initial weights of the plurality of initial weights.

### Claim 4 (depends on 3)

The method of claim 3, wherein extracting the (i+1)th plurality of feature maps comprises:
generating an (i+1)th plurality of filtered feature maps by applying an ith plurality of filters on the ith plurality of feature maps, the ith plurality of filters associated with the ith convolutional layer;
generating an (i+1)th plurality of normalized feature maps by applying a batch normalization process associated with the ith convolutional layer on the (i+1)th plurality of filtered feature maps, each normalized feature map of the (i+1)th plurality of normalized feature maps associated with a respective filtered feature map of the (i+1)th plurality of filtered feature maps;
generating an (i+1)th plurality of non-linear feature maps by implementing an ith non-linear activation function associated with the ith convolutional layer on each of the (i+1)th plurality of normalized feature maps; and
generating the (i+1)th plurality of feature maps by applying a max pooling associated with the ith convolutional layer on each of the (i+1)th plurality of non-linear feature maps.

### Claim 5 (depends on 4)

The method of claim 4, wherein obtaining the (i+1)th feature set comprises applying a global average pooling on the (i+1)th plurality of non-linear feature maps, the global average pooling comprising obtaining an average of each non-linear feature map of the (i+1)th plurality of non-linear feature maps.

### Claim 6 (depends on 4)

The method of claim 4, wherein applying the ith plurality of filters on the ith plurality of feature maps comprises applying a separable convolution on the ith plurality of feature maps.

### Claim 7 (depends on 4)

The method of claim 4, wherein
applying the (k−1)th fully connected output to the kth fully connected layer comprises implementing an (N+k)th non-linear activation function on the (k−1)th fully connected output, an (N+L)th non-linear activation function comprising a softmax function.

### Claim 8 (depends on 7)

The method of claim 7, wherein:
extracting the (i+1)th plurality of training feature maps comprises applying a first dropout process on the output of the ith convolutional layer; and
extracting the training distribution comprises applying a second dropout process on an output of the kth fully connected layer.

### Claim 9 (depends on 7)

The method of claim 7, wherein implementing each of the ith non-linear activation function and the (N+k)th non-linear activation function comprises implementing one of a rectified linear unit (ReLU) function or an exponential linear unit (ELU) function.

### Claim 10 (depends on 3)

The method of claim 3, wherein repeating the iterative process comprises repeating the iterative process until a value of the loss function becomes lower than a predefined threshold, the loss function defined by the following:
L
⁡
(
p
,
q
)
=
∑
m
=
1
M
⁢
p
m
⁢
log
⁡
(
p
m
q
m
)
where:
L() is the loss function,
p is the training distribution,
q is a ground truth distribution associated with the training image,
pm is an mth component of the training distribution,
qm is an mth component of the ground truth distribution, and
M is a number of the plurality of scores.

### Claim 11

A convolutional neural network system for quality assessment of an image, comprising:
a memory having processor-readable instructions stored therein; and
one or more processors configured to access the memory and execute the processor-readable instructions, which, when executed by the one or more processors configures the one or more processors to perform a method, the method comprising:
generating a plurality of convolutional layers comprising:
a first convolutional layer configured to receive the image, the image comprising a first plurality of feature maps; and
an ith convolutional layer configured to extract an (i+1)th plurality of feature maps from an ith plurality of feature maps, where 1≤i≤N and N is a number of the plurality of convolutional layers;
generating a global average pooling layer configured to obtain an (i+1)th feature set of a plurality of feature sets from the (i+1)th plurality of feature maps by obtaining an average of each feature map of the (i+1)th plurality of feature maps; and
generating a plurality of fully connected layers configured to extract a score distribution for a plurality of scores associated with the image from the plurality of feature sets, the plurality of fully connected layers comprising:
a first fully connected layer configured to generate a first fully connected output of a plurality of fully connected outputs from the plurality of feature sets; and
a kth fully connected layer configured to generate a kth fully connected output of the plurality of fully connected outputs from a (k−1)th fully connected output of the plurality of fully connected outputs by applying the (k−1)th fully connected output to an (N+k)th non-linear activation function where 1≤k≤L, where:
L is a number of the plurality of fully connected layers,
a 0th fully connected output comprises the plurality of feature sets, and
an Lth fully connected output of the plurality of fully connected outputs comprises the score distribution.

### Claim 12 (depends on 11)

The convolutional neural network system of claim 11, wherein the ith convolutional layer comprises:
an ith plurality of filters configured to generate an (i+1)th plurality of filtered feature maps from the ith plurality of feature maps by applying a separable convolution on the ith plurality of feature maps;
a batch normalization layer configured to generate an (i+1)th plurality of normalized feature maps from the (i+1)th plurality of filtered feature maps by applying a batch normalization process on the (i+1)th plurality of filtered feature maps, each normalized feature map of the (i+1)th plurality of normalized feature maps associated with a respective filtered feature map of the (i+1)th plurality of filtered feature maps;
an ith non-linear activation layer configured to generate an (i+1)th plurality of non-linear feature maps from the (i+1)th plurality of normalized feature maps by implementing an ith non-linear activation function on each of the (i+1)th plurality of non-linear feature maps, the ith non-linear activation function comprising one of a rectified linear unit (ReLU) function or an exponential linear unit (ELU) function; and
a pooling layer configured to generate the (i+1)th plurality of feature maps by apply a max pooling on each of the (i+1)th plurality of non-linear feature maps.

### Claim 13 (depends on 11)

The convolutional neural network system of claim 11, wherein
an (N+L)th non-linear activation function comprises a softmax function.

### Claim 14

A system for quality assessment of an image, the system comprising:
a memory having processor-readable instructions stored therein; and
one or more processors configured to access the memory and execute the processor-readable instructions, which, when executed by the one or more processors configures the one or more processors to perform a method, the method comprising:
generating an (i+1)th plurality of feature maps from the image by:
feeding the image to a first convolutional layer of a plurality of convolutional layers associated with a convolutional neural network, the image comprising a first plurality of feature maps; and
extracting the (i+1)th plurality of feature maps by applying an ith plurality of feature maps to an ith convolutional layer of the plurality of convolutional layers, where 1≤i≤N and N is a number of the plurality of convolutional layers;
obtaining an (i+1)th feature set of a plurality of feature sets from the (i+1)th plurality of feature maps applying a global average pooling on the (i+1)th plurality of feature maps, the global average pooling comprising obtaining an average of each feature map of the (i+1)th plurality of feature maps; and
extracting a score distribution for a plurality of scores associated with the image from the plurality of feature sets by:
generating a first fully connected output of a plurality of fully connected outputs by applying the plurality of feature sets to a first fully connected layer of a plurality of fully connected layers associated with the convolutional neural network; and
generating a kth fully connected output of the plurality of fully connected outputs by applying a (k−1)th fully connected output of a plurality of fully connected outputs to a kth fully connected layer of the plurality of fully connected layers,
where 1≤k≤L, L is a number of the plurality of fully connected layers, a 0th fully connected output comprises the plurality of feature sets, and an Lth fully connected output of the plurality of fully connected outputs comprises the score distribution.

### Claim 15 (depends on 14)

The system of claim 14, wherein generating the (i+1)th plurality of feature maps comprises training the convolutional neural network prior to feeding the image to the first convolutional layer, training the convolutional neural network comprising:
initializing the convolutional neural network by a plurality of initial weights; and
repeating an iterative process until a termination condition is satisfied, the iterative process comprising:
generating a plurality of updated weights by feeding a training image of a plurality of training images to the convolutional neural network; and
replacing the plurality of initial weights with the plurality of updated weights.

### Claim 16 (depends on 15)

The method of claim 15, wherein generating the plurality of updated weights comprises:
generating an (i+1)th plurality of training feature maps from the training image by:
feeding the training image to the first convolutional layer of the plurality of convolutional layers; and
extracting the (i+1)th plurality of training feature maps from an output of the ith convolutional layer;
obtaining an (i+1)th training feature set of the plurality of training feature sets from the (i+1)th plurality of training feature maps;
extracting a training distribution from the plurality of training feature sets by feeding the plurality of training feature sets to the first fully connected layer of the plurality of fully connected layers;
generating a plurality of adjustment values by minimizing a loss function, each of the plurality of adjustment values associated with a respective initial weight of the plurality of initial weights; and
obtaining the plurality of updated weights by adding each of the plurality of adjustment values to a respective initial weight of the plurality of initial weights.

### Claim 17 (depends on 16)

The system of claim 16, wherein extracting the (i+1)th plurality of feature maps comprises:
generating an (i+1)th plurality of filtered feature maps by applying an ith plurality of filters associated with the ith convolutional layer on the ith plurality of feature maps, applying the ith plurality of filters comprising applying a separable convolution on the ith plurality of feature maps;
generating an (i+1)th plurality of normalized feature maps by applying a batch normalization process associated with the ith convolutional layer on the (i+1)th plurality of filtered feature maps, each normalized feature map of the (i+1)th plurality of normalized feature maps associated with a respective filtered feature map of the (i+1)th plurality of filtered feature maps;
generating an (i+1)th plurality of non-linear feature maps by implementing an ith non-linear activation function associated with the ith convolutional layer on each of the (i+1)th plurality of normalized feature maps, the ith non-linear activation function comprising one of a rectified linear unit (ReLU) function or an exponential linear unit (ELU) function; and
generating the (i+1)th plurality of feature maps by applying a max pooling associated with the ith convolutional layer on each of the (i+1)th plurality of non-linear feature maps.

### Claim 18 (depends on 17)

The system of claim 17, wherein
applying the (k−1)th fully connected output to the kth fully connected layer comprises implementing an (N+k)th non-linear activation function on the (k−1)th fully connected output, an (N+L)th non-linear activation function comprising a softmax function.

### Claim 19 (depends on 18)

The system of claim 18, wherein:
extracting the (i+1)th plurality of training feature maps comprises applying a first dropout process on the output of the ith convolutional layer; and
extracting the training distribution comprises applying a second dropout process on an output of the kth fully connected layer.

### Claim 20 (depends on 16)

The system of claim 16, wherein repeating the iterative process comprises repeating the iterative process until a value of the loss function becomes lower than a predefined threshold, the loss function defined by the following:
L
⁡
(
p
,
q
)
=
∑
m
=
1
M
⁢
p
m
⁢
log
⁡
(
p
m
q
m
)
where:
L() is the loss function,
p is the training distribution,
q is a ground truth distribution associated with the training image,
pm is an mth component of the training distribution,
qm is an mth component of the ground truth distribution, and
M is a number of the plurality of scores.