Joint facial feature extraction and facial image quality estimation using a deep neural network (DNN) trained with a custom-labeled training dataset and having a common DNN backbone

A method performed by one or more processing resources of a computer system, the method comprising:

• receiving an image;
• predicting a suitability of the image for performing a facial feature extraction algorithm on the image by performing a quality prediction algorithm on the image;
• when the suitability is greater than a predetermined quality threshold, extracting facial features from a face contained within the image by applying the facial feature extraction algorithm;
• wherein the quality prediction algorithm and the facial feature extraction algorithm are jointly performed by a deep neural network (DNN) that has been trained by:
• training a common DNN backbone and the facial feature extraction algorithm based on a first training dataset including a plurality of training images, wherein the training the common DNN backbone and the facial feature extraction algorithm based on the first training data set yields an intermediate output of the common DNN backbone that is correlated to facial quality;
• and
• training the quality prediction algorithm based on a second training dataset and the intermediate output of the common DNN backbone while holding fixed the common DNN backbone;
• wherein the facial feature extraction algorithm and the quality prediction algorithm share a common DNN backbone of the DNN;
• wherein a first category of the plurality of categories includes a first subset of the example images representative of those of the example images for which the facial feature extraction algorithm cannot be performed;
• wherein a second category of the plurality of categories includes a second subset of the example images representative of those of the example images that are ideal for the facial feature extraction algorithm;
• and
• wherein a third category of the plurality of categories includes a third subset of the example images representative of those of the example images having a suitability for the facial feature extraction algorithm between that of the first category and the second category.

A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a computer system, causes the one or more processing resources to perform a method comprising:

• training a deep neural network (DNN) implementing a facial feature extraction algorithm and a quality prediction algorithm that share a common DNN backbone of the DNN to jointly perform (i) facial feature extraction in accordance with the facial feature extraction algorithm and (ii) a quality score in accordance with the quality prediction algorithm by:
• training the common DNN backbone and the facial feature extraction algorithm based on a first training dataset including a plurality of training images, wherein the training the common DNN backbone and the facial feature extraction algorithm based on the first training data set yields an intermediate output of the common DNN backbone that is correlated to facial quality;
• and
• training the quality prediction algorithm based on a second training dataset and the intermediate output of the common DNN backbone while holding fixed the common DNN backbone, wherein the second training dataset includes example images each labeled with a score value associated with a particular category of a plurality of categories, wherein a first category of the plurality of categories includes a first subset of the example images representative of those of the example images for which a facial feature extraction algorithm cannot be performed, wherein a second category of the plurality of categories includes a second subset of the example images representative of those of the example images that are ideal for the facial feature extraction algorithm, and wherein a third category of the plurality of categories includes a third subset of the example images representative of those of the example images having a suitability for the facial feature extraction algorithm between that of the first category and the second category.

A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a computer system, causes the one or more processing resources to perform a method comprising:

• receiving an image;
• predicting a suitability of the image for performing a facial feature extraction algorithm on the image by performing a quality prediction algorithm on the image;
• when the suitability is greater than a predetermined quality threshold, extracting facial features from a face contained within the image by applying the facial feature extraction algorithm;
• wherein the quality prediction algorithm and the facial feature extraction algorithm are jointly performed by a deep neural network (DNN) that has been trained by:
• training a common DNN backbone and the facial feature extraction algorithm based on a first training dataset including a plurality of training images, wherein the training the common DNN backbone and the facial feature extraction algorithm based on the first training data set yields an intermediate output of the common DNN backbone that is correlated to facial quality;
• and
• training the quality prediction algorithm based on a second training dataset and the intermediate output of the common DNN backbone while holding fixed the common DNN backbone;
• wherein the facial feature extraction algorithm and the quality prediction algorithm share a common DNN backbone of the DNN;
• wherein a first category of the plurality of categories includes a first subset of the example images representative of those of the example images for which the facial feature extraction algorithm cannot be performed;
• wherein a second category of the plurality of categories includes a second subset of the example images representative of those of the example images that are ideal for the facial feature extraction algorithm;
• and
• wherein a third category of the plurality of categories includes a third subset of the example images representative of those of the example images having a suitability for the facial feature extraction algorithm between that of the first category and the second category.

A system comprising:

• a processing resource;
• a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to:
• train a deep neural network (DNN) implementing a facial feature extraction algorithm and a quality prediction algorithm that share a common DNN backbone of the DNN to jointly perform (i) facial feature extraction in accordance with the facial feature extraction algorithm and (ii) a quality score in accordance with the quality prediction algorithm by:
• training the common DNN backbone and the facial feature extraction algorithm based on a first training dataset including a plurality of training images, wherein the training the common DNN backbone and the facial feature extraction algorithm based on the first training data set yields an intermediate output of the common DNN backbone that is correlated to facial quality;
• and
• training the quality prediction algorithm based on a second training dataset and the intermediate output of the common DNN backbone while holding fixed the common DNN backbone, wherein the second training dataset includes example images each labeled with a score value associated with a particular category of a plurality of categories, wherein a first category of the plurality of categories includes a first subset of the example images representative of those of the example images for which a facial feature extraction algorithm cannot be performed, wherein a second category of the plurality of categories includes a second subset of the example images representative of those of the example images that are ideal for the facial feature extraction algorithm, and wherein a third category of the plurality of categories includes a third subset of the example images representative of those of the example images having a suitability for the facial feature extraction algorithm between that of the first category and the second category.

A system comprising:

• a processing resource;
• a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to:
• receive an image;
• predict a suitability of the image for performing a facial feature extraction algorithm on the image by performing a quality prediction algorithm on the image;
• when the suitability is greater than a predetermined quality threshold, extracting facial features from a face contained within the image by applying the facial feature extraction algorithm;
• wherein the quality prediction algorithm and the facial feature extraction algorithm are jointly performed by a deep neural network (DNN) that has been trained by:
• training a common DNN backbone and the facial feature extraction algorithm based on a first training dataset including a plurality of training images, wherein the training the common DNN backbone and the facial feature extraction algorithm based on the first training data set yields an intermediate output of the common DNN backbone that is correlated to facial quality;
• and
• training the quality prediction algorithm based on a second training dataset and the intermediate output of the common DNN backbone while holding fixed the common DNN backbone;
• wherein the facial feature extraction algorithm and the quality prediction algorithm share a common DNN backbone of the DNN;
• wherein a first category of the plurality of categories includes a first subset of the example images representative of those of the example images for which the facial feature extraction algorithm cannot be performed;
• wherein a second category of the plurality of categories includes a second subset of the example images representative of those of the example images that are ideal for the facial feature extraction algorithm;
• and
• wherein a third category of the plurality of categories includes a third subset of the example images representative of those of the example images having a suitability for the facial feature extraction algorithm between that of the first category and the second category.