# US 11,908,233 B2 · Normalization of facial images using deep neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,233 B2 - Document ID: US11908233B2 - Status: Granted - Kind: B2 - Application number: 17/343,575 - Filed: 2021-06-09 - Published: 2022-05-05 - Granted: 2024-02-20 - Earliest claimed: 2020-11-02 - Assignees: Pinscreen, Inc. - Applicants: Pinscreen, Inc. - Inventors: Koki Nagano; Huiwen Luo; Zejian Wang; Jaewoo Seo; Liwen Hu; Lingyu Wei; Hao Li - Web page: https://patentgrep.com/patent/US11908233B2 ## CPC - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/169: {Holistic features and representations, i.e. based on the facial image taken as a whole} - G06T 5/60: using machine learning, e.g. neural networks - G06T 5/80: Geometric correction - G06T 5/94: based on local image properties, e.g. for local contrast enhancement - G06V 10/764: using classification, e.g. of video objects - G06T 2207/10024: Color image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T 2207/30201: Face - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system, method, and apparatus for generating a normalization of a single two-dimensional image of an unconstrained human face. The system receives the single two-dimensional image of the unconstrained human face, generates an undistorted face based on the unconstrained human face by removing perspective distortion from the unconstrained human face via a perspective undistortion network, generates an evenly lit face based on the undistorted face by normalizing lighting of the undistorted face via a lighting translation network, and generates a frontalized and neutralized expression face based on the evenly lit face via an expression neutralization network. ## Claims ### Claim 1 A deep learning-based framework system for generating a normalization of a single two-dimensional image of an unconstrained human face, the system comprising a processor and memory, the processor executing instructions which cause the processor to: receive the single two-dimensional image of the unconstrained human face; use a perspective undistortion network to remove perspective distortion from the unconstrained human face to generate an undistorted face based on the unconstrained human face, wherein removing perspective distortion includes performing three-dimensional face fitting to the undistorted face to obtain a fitted three-dimensional mesh, spherical harmonics shading coefficients, and UV texture; use a lighting translation network to generate an evenly lit face based on the undistorted face by normalizing lighting of the undistorted face; and use an expression neutralization network to generate a frontalized and neutralized expression face based on the evenly lit face. ### Claim 2 (depends on 1) The system of claim 1, wherein the perspective undistortion network: predicts a dense two-dimensional flow based on the image and two-dimensional landmarks detected in the image to generate the undistorted face. ### Claim 3 (depends on 1) The system of claim 1, wherein the lighting translation network uses the spherical harmonics shading coefficients to generate the evenly lit face. ### Claim 4 (depends on 1) The system of claim 1, wherein the expression neutralization network uses the UV texture to generate the frontalized and neutralized expression face. ### Claim 5 (depends on 1) The system of claim 1, wherein the expression neutralization network generates a frontalized and neutralized face by: using a geometry neutralization network to neutralize geometry of the evenly lit face to generate a geometry neutralized face; and using a facial texture neutralization network to neutralize facial texture of the geometry neutralized face to generate the frontalized and neutralized expression face. ### Claim 6 (depends on 1) The system of claim 1, wherein the perspective undistortion network is a generative adversarial network trained via dense flow fields by fitting three-dimensional face models to input images with known focal length and camera-to-subject distance; wherein the lighting translation network is a conditional generative adversarial network trained via synthetic illumination data based on portrait relighting; and wherein the expression neutralization network is a generative adversarial network trained based on expression-to-neutral and neutral-to-expression datasets. ### Claim 7 A method of generating a normalization of a single two-dimensional image of an unconstrained human face comprising: receiving the single two-dimensional image of the unconstrained human face; using a perspective undistortion network to remove perspective distortion from the unconstrained human face to generate an undistorted face based on the unconstrained human face, wherein removing perspective distortion includes performing three-dimensional face fitting to the undistorted face to obtain a fitted three-dimensional mesh, spherical harmonics shading coefficients, and UV texture; using a lighting translation network to generate an evenly lit face based on the undistorted face by normalizing lighting of the undistorted face; and using an expression neutralization network to generate a frontalized and neutralized expression face based on the evenly lit face. ### Claim 8 (depends on 7) The method of claim 7, wherein the perspective undistortion network: predicts a dense two-dimensional flow based on the image and two-dimensional landmarks detected in the image to generate the undistorted face. ### Claim 9 (depends on 7) The method of claim 7, wherein the lighting translation network uses the spherical harmonics to generate the evenly lit face. ### Claim 10 (depends on 7) The method of claim 7, wherein the expression neutralization network uses the UV texture to generate the frontalized and neutralized expression face. ### Claim 11 (depends on 7) The method of claim 7, wherein the expression neutralization network generates a frontalized and neutralized face by: using a geometry neutralization network to neutralize geometry of the evenly lit face to generate a geometry neutralized face; and using a facial texture neutralization network to neutralize facial texture of the geometry neutralized face to generate the frontalized and neutralized face. ### Claim 12 (depends on 7) The method of claim 7, wherein the perspective undistortion network is a generative adversarial network trained via dense flow fields by fitting three-dimensional face models to input images with known focal length and camera-to-subject distance; wherein the lighting translation network is a conditional generative adversarial network trained via synthetic illumination data based on portrait relighting; and wherein the expression neutralization network is a generative adversarial network trained based on expression-to-neutral and neutral-to-expression datasets. ### Claim 13 An apparatus comprising a storage medium storing a program having instructions which when executed by a processor will cause the processor to: receive the single two-dimensional image of an unconstrained human face; use a perspective undistortion network to remove perspective distortion from the unconstrained human face to generate an undistorted face based on the unconstrained human face, wherein removing perspective distortion includes performing three-dimensional face fitting to the undistorted face to obtain a fitted three-dimensional mesh, spherical harmonics shading coefficients, and UV texture; use a lighting translation network to generate an evenly lit face based on the undistorted face by normalizing lighting of the undistorted face; and use an expression neutralization network to generate a frontalized and neutralized expression face based on the evenly lit face. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the perspective undistortion network: predicts a dense two-dimensional flow based on the image and two-dimensional landmarks detected in the image to generate the undistorted face. ### Claim 15 (depends on 13) The apparatus of claim 13, wherein the lighting translation network uses the spherical harmonics shading coefficients to generate the evenly lit face. ### Claim 16 (depends on 13) The apparatus of claim 13, wherein the expression neutralization network generates a frontalized and neutralized face by: using a geometry neutralization network to neutralize geometry of the evenly lit face to generate a geometry neutralized face; and using a facial texture neutralization network to neutralize facial texture of the geometry neutralized face to generate the frontalized and neutralized face. ### Claim 17 (depends on 13) The apparatus of claim 13, wherein the perspective undistortion network is a generative adversarial network trained via dense flow fields by fitting three-dimensional face models to input images with known focal length and camera-to-subject distance; wherein the lighting translation network is a conditional generative adversarial network trained via synthetic illumination data based on portrait relighting; and wherein the expression neutralization network is a generative adversarial network trained based on expression-to-neutral and neutral-to-expression datasets.