US 2021/0201150 A1 · FRAME SELECTION BASED ON A TRAINED NEURAL NETWORK

Abstract

Various embodiments describe frame selection based on training and using a neural network. In an example, the neural network is a convolutional neural network trained with training pairs. Each training pair includes two training frames from a frame collection. The loss function relies on the estimated quality difference between the two training frames. Further, the definition of the loss function varies based on the actual quality difference between these two frames. In a further example, the neural network is trained by incorporating facial heatmaps generated from the training frames and facial quality scores of faces detected in the training frames. In addition, the training involves using a feature mean that represents an average of the features of the training frames belonging to the same frame collection. Once the neural network is trained, a frame collection is input thereto and a frame is selected based on generated quality scores.

Claims20

Dependent claims

A computer-implemented method, comprising:

providing a collection of images to a neural network, the neural network trained using training data comprising training images and associated training labels, wherein a training label associated with a training image indicates a quality measure for the training image, the neural network having associated parameters resulting from minimization of a loss function based on a first quality difference and a second quality difference, wherein the first quality difference is based on a training label associated with a first training image in the training images and a training label associated with a second training image in the training images, and wherein the second quality difference is based on an estimation of quality of the first training image and an estimation of quality of the second training image generated by the neural network;
generating, using the neural network, an estimation of quality for each image in the collection of images;
and
based upon the estimations of quality generated for the collection of images, selecting and storing a particular image from the collection of images.

1 more element

A non-transitory computer-readable storage medium comprising instructions that, upon execution on a computing system, cause the computing system to perform operations comprising:

providing a collection of images to a neural network, the neural network trained using training data comprising training images and associated training labels, wherein a training label associated with a training image indicates a quality measure for the training image, the neural network having associated parameters resulting from minimization of a loss function based on a first quality difference and a second quality difference, wherein the first quality difference is based on a training label associated with a first training image in the training images and a training label associated with a second training image in the training images, and wherein the second quality difference is based on an estimation of quality of the first training image and an estimation of quality of the second training image generated by the neural network;
generating, using the neural network, an estimation of quality for each image in the collection of images;
and
based upon the estimations of quality generated for the collection of images, selecting and storing a particular image from the collection of images.

1 more element

A system comprising:

a processing device;
and
a non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured to execute program code stored in the non-transitory computer-readable medium and thereby perform operations comprising:
providing a collection of images to a neural network, the neural network trained using training data comprising training images and associated training labels, wherein a training label associated with a training image indicates a quality measure for the training image, the neural network having associated parameters resulting from minimization of a loss function based on a first quality difference and a second quality difference, wherein the first quality difference is based on a training label associated with a first training image in the training images and a training label associated with a second training image in the training images, and wherein the second quality difference is based on an estimation of quality of the first training image and an estimation of quality of the second training image generated by the neural network;
generating, using the neural network, an estimation of quality for each image in the collection of images;
and
based upon the estimations of quality generated for the collection of images, selecting and storing a particular image from the collection of images.

4 more elements

Dependencies are inferred from the claim text.

Frame selection based on a trained neural network

Life of this patent

Frame selection based on a trained neural network

Life of this patent

Abstract

Claims20