System and method for reconstruction of an animatable three-dimensional human head model from an image using an implicit representation network

A method for generating a three-dimensional (3D) model of a first face from an input two-dimensional (2D) facial image of the first face, the method comprising:

• training the hypernetwork (H) by:
• obtaining a plurality of ground truth images of a second face, each with a different view of the second face;
• providing one image of the plurality of ground truth images of the second face to the hypernetwork (H) to generate weights for the IRN;
• generating, by the IRN, a plurality of generated images of the second face, each with a view of the second face that corresponds to a view of the second face in one image of the plurality of ground truth images of a second face;
• calculating a first loss term for the plurality of generated images of the second face;
• reconstructing the 3D face model of the second face, by providing the plurality of generated images of the second face to the network trained to generate a 3D face model from the plurality of generated images of the second face;
• rendering, from the 3D face model of the second face, a plurality of rendered images of the second face, each with a view of the second face that corresponds to a view of the second face in one image of the plurality of ground truth images of a second face;
• calculating a second loss term for the plurality of rendered images of the second face;
• and
• using a loss function comprising the first and the second loss terms to train the hypernetwork (H);
• providing the input 2D facial image to a hypernetwork (H) to generate weights for an implicit representation network (IRN);
• generating, by the IRN with the generated weights, a plurality of facial images of the first face, each with a different view of the first face;
• and
• reconstructing the 3D model of the first face by providing the plurality of facial images of the first face to a network trained to generate a 3D face model from the plurality of facial images.

A system for reconstructing a three-dimensional (3D) face model from an input 2D image of the face, the system comprising:

• a memory;
• and
• a processor configured to:
• train the hypernetwork (H) by:
• obtaining ground truth multi-view images of a second face;
• feeding one image of the ground truth multi-view images of the second face into the hypernetwork (H) to generate weights for the IRN;
• generating, by the IRN, generated multi-view images of the second face;
• calculating a first loss term for the generated multi-view images of the second face;
• generating the 3D face model of the second face, by feeding the generated multi-view images into the network trained to generate a 3D face model from the generated multi-view face images;
• rendering, from the 3D face model of the second face, rendered multi-view images of the second face;
• calculating a second loss term for the rendered multi-view images of the second face;
• and
• using the first and the second loss terms to train the hypernetwork (H);
• feed the input 2D image of the first face into a hypernetwork (H) to generate weights for an implicit representation network (IRN);
• generate, by the IRN with the generated weights, multi-view face images of the first face;
• and
• generate the 3D model of the first face by feeding the multi-view face images of the first face into a network trained to generate a 3D face model from the multi-view face images.