US 12,597,096 B2 · Real-time facial restoration and relighting in videos using facial enhancement neural networks

Abstract

The present disclosure relates to an image restoration system that efficiently and accurately produces high-quality images captured under low-light and/or low-quality environmental conditions. To illustrate, when a user is in a low-lit environment and participating in a video stream, the image restoration system enhances the quality of the image by dynamically re-lighting the user's face. Moreover, it significantly enhances the image quality to the extent that other users viewing the video stream are unaware of the poor environmental conditions of the user. In addition, the image restoration system creates and utilizes an image restoration machine-learning model to improve the quality of low-quality images by re-lighting and restoring them in real time. Various implementations combine an autoencoder model with a distortion classifier model to create the image restoration machine-learning model.

Claims20

Dependent claims

A computer-implemented method comprising:

detecting a face within a digital image that includes the face and an image background;
generating a light-enhanced face image of the face utilizing an image restoration machine-learning model that includes an autoencoder model and a distortion classifier by:
producing a distortion classification by the distortion classifier;
and
concatenating the distortion classification with encoded feature vectors generated by an encoder of the autoencoder model;
generating an enhanced digital image by combining the light-enhanced face image with the image background;
and
providing the enhanced digital image for display on a computing device.

5 more elements

A system comprising:

an image restoration machine-learning model that includes a distortion classifier and an autoencoder model having an encoder and a generator, wherein the distortion classifier is separate from the autoencoder model;
a processor;
and
a computer memory comprising instructions that, when executed by the processor, cause the system to carry out operations comprising:
detecting a face within a digital image that includes the face and an image background;
generating a light-enhanced face image of the face utilizing the image restoration machine-learning model based on:
producing a distortion classification by the distortion classifier;
combining the distortion classification from the distortion classifier with encoded feature vectors output from the encoder of the autoencoder model to generate a combined output;
and
using the combined output as an input to the generator of the autoencoder model;
and
generating an enhanced digital image by combining the light-enhanced face image with the image background.

9 more elements

Dependencies are inferred from the claim text.

Real-time facial restoration and relighting in videos using facial enhancement neural networks

Life of this patent

Real-time facial restoration and relighting in videos using facial enhancement neural networks

Life of this patent

Abstract

Claims20