3D face modeling based on neural networks

A method, comprising:

• receiving a two-dimensional (2D) color image of a human face with a first face expression;
• obtaining a first three-dimensional (3D) mesh of the human face with the first face expression, based on the received 2D color image;
• generating first texture information for the obtained first 3D mesh based on the received 2D color image;
• generating a first set of displacement maps based on the obtained first 3D mesh, wherein
• the first set of displacement maps comprises a first displacement map along a first orthogonal axis, a second displacement map along a second orthogonal axis, and a third displacement map along a third orthogonal axis, and
• each of the first set of displacement maps comprises geometric information associated with vertices on a surface of the first 3D mesh along a corresponding orthogonal axis;
• obtaining a mean 3D mesh of the human face with the first face expression, wherein
• the geometric information of the first displacement map includes a difference between a vertex position on the first 3D mesh along the first orthogonal axis and a corresponding vertex position on the mean 3D mesh along the first orthogonal axis;
• feeding an input comprising the generated first texture information and the generated first set of displacement maps to a neural network, which is trained on an image-to-image translation task;
• receiving an output of the neural network for the fed input, wherein the received output comprises second texture information and a second set of displacement maps;
• deforming the obtained first 3D mesh of the human face based on UV mapping the second set of displacement maps onto the first 3D mesh;
• and
• generating, based on the deformed first 3D mesh of the human face, a second 3D mesh of the human face with a second face expression which is different from the first face expression.

A non-transitory computer-readable storage medium configured to store instructions that, in response to being executed, causes an electronic device to perform operations, the operations comprising:

• receiving a two-dimensional (2D) color image of a human face with a first face expression;
• obtaining a first three-dimensional (3D) mesh of the human face with the first face expression, based on the received 2D color image;
• generating first texture information for the obtained first 3D mesh based on the received 2D color image;
• generating a first set of displacement maps based on the obtained first 3D mesh, wherein
• the first set of displacement maps comprises a first displacement map along a first orthogonal axis, a second displacement map along a second orthogonal axis, and a third displacement map along a third orthogonal axis, and
• each of the first set of displacement maps comprises geometric information associated with vertices on a surface of the first 3D mesh along a corresponding orthogonal axis;
• obtaining a mean 3D mesh of the human face with the first face expression, wherein
• the geometric information of the first displacement map includes a difference between a vertex position on the first 3D mesh along the first orthogonal axis and a corresponding vertex position on the mean 3D mesh along the first orthogonal axis;
• feeding an input comprising the generated first texture information and the generated first set of displacement maps to a neural network, which is trained on an image-to-image translation task;
• receiving an output of the neural network for the fed input, wherein the received output comprises second texture information and a second set of displacement maps;
• deforming the obtained first 3D mesh of the human face based on UV mapping the second set of displacement maps onto the first 3D mesh;
• and
• generating, based on the deformed first 3D mesh of the human face, a second 3D mesh of the human face with a second face expression which is different from the first face expression.