# US 10,872,457 B1 · Facial texture map generation using single color image and depth information > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,457 B1 - Document ID: US10872457B1 - Status: Granted - Kind: B1 - Application number: 16/515,263 - Filed: 2019-07-18 - Granted: 2020-12-22 - Earliest claimed: 2019-07-18 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Jie Hu; Mohammad Gharavi-Alkhansari - Web page: https://patentgrep.com/patent/US10872457B1 ## CPC - G06T 15/04: Texture mapping - G06T 3/02: Affine transformations (for image registration G06T3/147; for image mosaicing G06T3/4038) - G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529) - G06T 15/506: {Illumination models} - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30201: Face - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An electronic apparatus for facial texture map generation using single color image and depth information is provided. The electronic apparatus stores a first color image and corresponding depth information of a face, an un-textured three-dimensional (3D) face model of the face and a reference facial texture model. The electronic apparatus generates a first texture map of the face based on the un-textured 3D face model, the first color image, and the depth information. The electronic apparatus further generates a second texture map by texture model fitting of the reference facial texture model on a plurality of first effective points on the first texture map. The electronic apparatus refines the second texture map based on minimization of an objective function for the second texture map and further generates a final texture map based on the first texture map and the refined second texture map. ## Claims ### Claim 1 An electronic apparatus, comprising: circuitry configured to: generate a first texture map of a face of a user based on an un-textured 3D face model, a first color image, and corresponding depth information of the face of the user; generate a second texture map of the face of the user by a texture model fitting process of a reference facial texture model on a plurality of first effective points on the generated first texture map, wherein the reference facial texture model is a Principal Component Analysis (PCA) model of a plurality of facial texture maps for a plurality of faces different from the face of the user, and the plurality of first effective points corresponds to points that belong to the face of the user observed in the first color image; refine the generated second texture map based on minimization of an objective function for the generated second texture map; extract a first portion of the first texture map based on a first mask for the face of the user; extract a second portion of the refined second texture map based on a second mask for the face of the user; and generate a final texture map for the face of the user based on a combination of the extracted first portion and the extracted second portion. ### Claim 2 (depends on 1) The electronic apparatus according to claim 1, wherein the un-textured 3D face model is a shape-refined 3D face model of the face of the user. ### Claim 3 (depends on 1) The electronic apparatus according to claim 1, wherein the PCA model corresponds to a combination of eigenvectors and eigenvalues that are extracted by application of PCA on the plurality of facial texture maps for the plurality of faces different from the face of the user. ### Claim 4 (depends on 1) The electronic apparatus according to claim 1, further comprising a scanning device configured to capture a first plurality of color images of the face of the user and first depth information corresponding to the first plurality of color images, wherein the first plurality of color images and the corresponding first depth information includes the first color image and the corresponding depth information. ### Claim 5 (depends on 4) The electronic apparatus according to claim 4, wherein the circuitry is further configured to generate the un-textured 3D face model of the face of the user based on the captured first plurality of color images and corresponding first depth information. ### Claim 6 (depends on 1) The electronic apparatus according to claim 1, further comprising a scanning device configured to capture a second plurality of color images and corresponding second depth information of the plurality of faces different from the face of the user, wherein the captured second plurality of color images of the plurality of faces are captured under a common ambient lighting condition. ### Claim 7 (depends on 6) The electronic apparatus according to claim 6, wherein the circuitry is further configured to: generate a face mesh of each of the plurality of faces based on the captured second plurality of color images, wherein a first face mesh of a first face of the plurality of faces is aligned with a second face mesh of a second face of the plurality of faces; and generate each facial texture map of the plurality of facial texture maps for a corresponding face of the plurality of faces based on unwrap of the generated face mesh of the corresponding face of the plurality of faces on a U-V coordinate map. ### Claim 8 (depends on 7) The electronic apparatus according to claim 7, wherein the circuitry is further configured to generate the reference facial texture model by application of Principal Component Analysis (PCA) on the plurality of facial texture maps for the plurality of faces different from the face of the user. ### Claim 9 (depends on 1) The electronic apparatus according to claim 1, wherein the circuitry is further configured to: determine a first plurality of triangles on the first color image; and map each triangle of the first plurality of triangles on the first color image to a corresponding triangle of a second plurality of triangles on the un-textured 3D face model, based on a perspective projection of the first color image on the un-textured 3D face model in a 3D space. ### Claim 10 (depends on 9) The electronic apparatus according to claim 9, wherein the circuitry is further configured to determine a third plurality of triangles on a U-V coordinate map, by a U-V coordinate mapping process of the second plurality of triangles on the U-V coordinate map. ### Claim 11 (depends on 10) The electronic apparatus according to claim 10, wherein the circuitry is further configured to: compute an affine transformation between the third plurality of triangles on the U-V coordinate map and the first plurality of triangles on the first color image; and determine a textured U-V coordinate map by application of texture to each triangle of the third plurality of triangles, wherein the application of texture to each triangle is based on the computed affine transformation, and the textured U-V coordinate map is the first texture map. ### Claim 12 (depends on 11) The electronic apparatus according to claim 11, wherein the application of the texture to each triangle of the third plurality of triangles on the U-V coordinate map is further based on interpolation of color information from a corresponding triangle of the first plurality of triangles on the first color image. ### Claim 13 (depends on 1) The electronic apparatus according to claim 1, wherein the circuitry is further configured to iteratively estimate a lighting strength for each of ambient light and diffused light, a lighting direction, and a specular reflectance for the first texture map, based on a reflectance model, the first texture map, and the un-textured 3D face model, and the reflectance model comprises a relationship between the lighting strength for each of the ambient light and the diffused light, the lighting direction, and the specular reflectance. ### Claim 14 (depends on 13) The electronic apparatus according to claim 13, wherein in the texture model fitting process, the circuitry is configured to estimate a set of albedo values for a plurality of points on the first texture map, based on the reference facial texture model, the estimated lighting strength for each of the ambient light and the diffused light, the lighting direction, and the specular reflectance. ### Claim 15 (depends on 1) The electronic apparatus according to claim 1, wherein each effective point of the plurality of first effective points on the generated first texture map is a point for which a corresponding point is observed from the first color image. ### Claim 16 (depends on 1) The electronic apparatus according to claim 1, wherein the circuitry is further configured to: compute an offset term for a plurality of points on the second texture map based on a first relationship for a first plurality of correspondences between a plurality of points on the first texture map and the plurality of points on the second texture map; and compute a smoothening term for the plurality of points on the second texture map based on a second relationship for a second plurality of correspondences between neighboring points of the plurality of points on the second texture map. ### Claim 17 (depends on 16) The electronic apparatus according to claim 16, wherein the circuitry is further configured to minimize the objective function based on the computed offset term and the computed smoothening term. ### Claim 18 (depends on 17) The electronic apparatus according to claim 17, wherein the circuitry is further configured to refine the second texture map further based on a removal of a lighting influence from the generated first texture map. ### Claim 19 (depends on 18) The electronic apparatus according to claim 18, wherein the circuitry is further configured to add the lighting influence of the first texture map to the refined second texture map. ### Claim 20 (depends on 1) The electronic apparatus according to claim 1, wherein the first portion corresponds to a front face portion of the face of the user, and the second portion corresponds to a remnant face portion of the face of the user. ### Claim 21 (depends on 1) The electronic apparatus according to claim 1, wherein the circuitry is further configured to generate a textured 3D face model of the face of the user by rendering the generated final texture map onto the un-textured 3D face model. ### Claim 22 A method, comprising: in an electronic apparatus that comprises a memory and circuitry: generating a first texture map of a face of a user based on an un-textured three-dimensional (3D) face model, a first color image, and corresponding depth information of the face of the user; generating a second texture map of the face of the user by texture model fitting of a reference facial texture model on a plurality of first effective points on the generated first texture map, wherein the reference facial texture model is a Principal Component Analysis (PCA) model of a plurality of facial texture maps for a plurality of faces different from the face of the user, and the plurality of first effective points corresponds to points that belong to the face of the user observed in the first color image; refining the generated second texture map based on minimization of an objective function for the generated second texture map; extracting a first portion of the first texture map based on a first mask for the face of the user; extracting a second portion of the refined second texture map based on a second mask for the face of the user; and generating a final texture map for the face of the user based on a combination of the extracted first portion and the extracted second portion. ### Claim 23 An electronic apparatus, comprising: circuitry configured to: generate a first texture map of a face of a user based on an un-textured 3D face model, a first color image, and corresponding depth information of the face of the user; generate a second texture map of the face of the user by a texture model fitting process of a reference facial texture model on a plurality of first effective points on the generated first texture map, wherein the reference facial texture model is a Principal Component Analysis (PCA) model of a plurality of facial texture maps for a plurality of faces different from the face of the user, and the plurality of first effective points corresponds to points that belong to the face of the user observed in the first color image; compute an offset term for a plurality of points on the second texture map based on a first relationship for a first plurality of correspondences between a plurality of points on the first texture map and the plurality of points on the second texture map; compute a smoothening term for the plurality of points on the second texture map based on a second relationship for a second plurality of correspondences between neighboring points of the plurality of points on the second texture map; refine the generated second texture map, based on a minimization of an objective function for the generated second texture map, wherein the objective function includes the computed offset term and computed smoothening term; and generate a final texture map for face of the user based on the generated first texture map and the refined second texture map.