Selfie perspective undistortion by 3D face GAN inversion

A method of image processing using a pre-trained three-dimension (3D) face generative adversarial network (GAN) having a generator and a neural renderer, comprising the steps of:

• processing an input image including a face;
• improving face latent code and camera parameters of the 3D face GAN by:
• freezing the generator and the neural renderer;
• obtaining a depth map and a normal map, and rendering photorealistic face images based on a random face latent and initialization of the camera parameters;
• calculating losses;
• and
• using gradient descent to improve the face latent code and camera parameters until the losses converge;
• fine tuning the generator;
• and
• generating an improved image of the face with reduced face distortion by setting a longer camera-to-face distance, wherein a face-to-camera distance is estimated by a known iris size of the face, wherein the estimated face-to-camera distance is used to improve the face latent code and camera parameters.

A method of image processing using a pre-trained three-dimension (3D) face generative adversarial network (GAN) having a generator and a neural renderer, comprising the steps of:

• processing an input image including a face;
• improving face latent code and camera parameters of the 3D face GAN by:
• freezing the generator and the neural renderer;
• obtaining a depth map and a normal map, and rendering photorealistic face images based on a random face latent and initialization of the camera parameters;
• calculating losses;
• and
• using gradient descent to improve the face latent code and camera parameters until the losses converge;
• fine tuning the generator;
• and
• generating an improved image of the face with reduced face distortion by setting a longer camera-to-face distance, wherein a face-to-camera distance is estimated by a known pupillary distance of the face, wherein the estimated face-to-camera distance is used to improve the face latent code and camera parameters.

A pre-trained three-dimension (3D) face generative adversarial network (GAN) having a generator and a neural renderer, configured to:

• process an input image including a face;
• improve face latent code and camera parameters of the 3D face GAN by:
• freezing the generator and the neural renderer to improve face latent code and camera parameters;
• adjusting a face-to-camera distance and a camera focal length while maintaining a pupillary distance of the face constant;
• and
• rendering a photorealistic image;
• fine tune the generator;
• and
• generate an improved image of the face with reduced face distortion due to setting a longer camera-to-face distance.

A non-transitory computer readable storage medium that stores instructions that when executed by a processor cause the processor to process an image using a pre-trained three-dimension (3D) face generative adversarial network (GAN) having a generator and a neural renderer by performing the steps of:

• processing the image including a face;
• improving face latent code and camera parameters of the 3D face GAN by;
• freezing the generator and the neural renderer to improve face latent code and camera parameters;
• adjusting a face-to-camera distance and a camera focal length while maintaining a pupillary distance of the face constant;
• and
• rendering a photorealistic image;
• tuning a generator of the 3D face GAN;
• and
• generating an improved image of the face with reduced face distortion due to a longer camera-to-face distance.