Facial texture map generation using single color image and depth information

A method, comprising:

• in an electronic apparatus that comprises a memory and circuitry:
• generating a first texture map of a face of a user based on an un-textured three-dimensional (3D) face model, a first color image, and corresponding depth information of the face of the user;
• generating a second texture map of the face of the user by texture model fitting of a reference facial texture model on a plurality of first effective points on the generated first texture map, wherein
• the reference facial texture model is a Principal Component Analysis (PCA) model of a plurality of facial texture maps for a plurality of faces different from the face of the user, and
• the plurality of first effective points corresponds to points that belong to the face of the user observed in the first color image;
• refining the generated second texture map based on minimization of an objective function for the generated second texture map;
• extracting a first portion of the first texture map based on a first mask for the face of the user;
• extracting a second portion of the refined second texture map based on a second mask for the face of the user;
• and
• generating a final texture map for the face of the user based on a combination of the extracted first portion and the extracted second portion.

An electronic apparatus, comprising:

• circuitry configured to:
• generate a first texture map of a face of a user based on an un-textured 3D face model, a first color image, and corresponding depth information of the face of the user;
• generate a second texture map of the face of the user by a texture model fitting process of a reference facial texture model on a plurality of first effective points on the generated first texture map, wherein
• the reference facial texture model is a Principal Component Analysis (PCA) model of a plurality of facial texture maps for a plurality of faces different from the face of the user, and
• the plurality of first effective points corresponds to points that belong to the face of the user observed in the first color image;
• compute an offset term for a plurality of points on the second texture map based on a first relationship for a first plurality of correspondences between a plurality of points on the first texture map and the plurality of points on the second texture map;
• compute a smoothening term for the plurality of points on the second texture map based on a second relationship for a second plurality of correspondences between neighboring points of the plurality of points on the second texture map;
• refine the generated second texture map, based on a minimization of an objective function for the generated second texture map, wherein the objective function includes the computed offset term and computed smoothening term;
• and generate a final texture map for face of the user based on the generated first texture map and the refined second texture map.