# US 10,482,656 B2 · 3D face modeling methods and apparatuses > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,656 B2 - Document ID: US10482656B2 - Status: Granted - Kind: B2 - Application number: 15/344,903 - Filed: 2016-11-07 - Published: 2017-06-01 - Granted: 2019-11-19 - Earliest claimed: 2015-12-01 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Seon Min Rhee; Jungbae Kim; Jaejoon Han - Web page: https://patentgrep.com/patent/US10482656B2 ## CPC - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 15/04: Texture mapping - G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes - G06V 10/426: Graphical representations - G06V 10/7557: {based on appearance, e.g. active appearance models [AAM]} - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A three-dimensional (3D) face modeling method and apparatus is disclosed. The 3D face modeling apparatus may generate a personalized 3D face model using a two-dimensional (2D) input image and a generic 3D face model, obtain a depth image and a texture image using the generated personalized 3D face model, determine a patch region of each of the depth image and the texture image, and adjust a shape of the personalized 3D face model based on a matching relationship between the patch region of the depth image and the patch region of the texture image. ## Claims ### Claim 1 A three-dimensional (3D) face modeling method, comprising: generating a personalized 3D face model using a two-dimensional (2D) input image and a generic 3D face model; obtaining a depth image and a texture image using the personalized 3D face model; determining a patch region of the depth image and a patch region of the texture image; and adjusting a shape of the personalized 3D face model based on a relationship between the patch region of the depth image and the patch region of the texture image, wherein the obtaining includes, obtaining the depth image by projecting, onto a 2D plane, a depth value of the personalized 3D face model, and obtaining the texture image by projecting, onto a 2D plane, a 3D texture face model generated by mapping a texture of the 2D input image to the personalized 3D face model, wherein the adjusting includes, determining a projection location at which a current vertex of the personalized 3D face model is projected to the patch region of the depth image; determining a point in the patch region of the texture image corresponding to the projection location, determining 3D coordinates of the point based on a depth corresponding to the projection location and a normal direction corresponding to the patch region of the texture image, and adjusting the shape of the personalized 3D face model based on the determined 3D coordinates of the point. ### Claim 2 (depends on 1) The method of claim 1, wherein the determining of the point comprises: determining a transformation matrix based on locations of landmarks forming the patch region of the depth image and locations of landmarks forming the patch region of the texture image; and determining the point by applying the determined transformation matrix to the projection location. ### Claim 3 (depends on 1) The method of claim 1, wherein the depth corresponding to the projection location is based on a normal direction towards a current vertex of the personalized 3D face model, the projection location being a location at which the current vertex is projected to the patch region of the depth image. ### Claim 4 (depends on 1) The method of claim 1, wherein the determining of the 3D coordinates of the point comprises: determining a location of the point as the 3D coordinates, the location being moved by the depth corresponding to the projection location in a normal direction of a plane in a 3D space corresponding to the patch region of the texture image. ### Claim 5 (depends on 1) The method of claim 1, wherein the determining of the patch region of the texture image comprises: detecting facial landmarks in the texture image; and determining the patch region of the texture image based on the facial landmarks detected in the texture image. ### Claim 6 (depends on 1) The method of claim 1, wherein the determining of the patch region of the depth image comprises: detecting first landmarks in the depth image based on locations in the depth image, second landmarks in the personalized 3D face model are projected to the locations; and determining the patch region of the depth image based on the first landmarks detected in the depth image. ### Claim 7 (depends on 1) The method of claim 1, wherein the generating of the personalized 3D face model comprises: detecting facial landmarks in the 2D input image; estimating a parameter to be applied to the generic 3D face model based on locations of the facial landmarks; and obtaining the personalized 3D face model by applying the estimated parameter to the generic 3D face model. ### Claim 8 (depends on 1) The method of claim 1, wherein the obtaining of the depth image comprises: obtaining the depth image based on a depth value of a facial shape of the personalized 3D face model. ### Claim 9 (depends on 1) The method of claim 1, wherein the obtaining of the texture image comprises: projecting the personalized 3D face model to which the texture is mapped onto a 2D plane. ### Claim 10 (depends on 1) A non-transitory computer-readable medium comprising program code that, when executed by a processor, causes the processor to perform the method of claim 1. ### Claim 11 (depends on 1) The method of claim 1, further comprising: outputting the adjusted personalized 3D face model. ### Claim 12 (depends on 1) The method of claim 1, wherein the projection location and the point in the patch region of the texture image have different coordinates. ### Claim 13 (depends on 12) The method of claim 12, wherein the point in the patch region has 2D coordinates and the determining the 3D coordinates includes applying the depth to the 2D coordinates to determine the 3D coordinates. ### Claim 14 (depends on 1) The method of claim 1, wherein the determining the point in the patch region of the texture image corresponding to the projection location determines the point in the patch region of the texture image corresponding to the projection location using an affine transformation. ### Claim 15 A three-dimensional (3D) face modeling apparatus, comprising: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to cause the apparatus to, generate a personalized 3D face model using a two-dimensional (2D) input image and a generic 3D face model; obtain a depth image and a texture image using the generated personalized 3D face model; determine a patch region of the depth image and a patch region of the texture image; and adjust a shape of the personalized 3D face model based on a matching relationship between the patch region of the depth image and the patch region of the texture image, wherein the processor is configured to execute the instructions to cause the apparatus to, obtain the depth image by projecting, onto a 2D plane, a depth value of the personalized 3D face model, and obtain the texture image by projecting, onto a 2D plane, a 3D texture face model generated by mapping a texture of the 2D input image to the personalized 3D face model, determine a projection location at which a current vertex of the personalized 3D face model is projected to the patch region of the depth image; determine a point in the patch region of the texture image corresponding to the projection location, determine 3D coordinates of the point based on a depth corresponding to the projection location and a normal direction corresponding to the patch region of the texture image, and adjust the shape of the personalized 3D face model based on the determined 3D coordinates of the point. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the processor is configured to execute the instructions to cause the apparatus to, determine a location of the point as the 3D coordinates, the location being moved by the depth corresponding to the projection location in a normal direction of the patch region of the texture image. ### Claim 17 (depends on 15) The apparatus of claim 15, wherein the patch region of the texture image is based on facial landmarks detected in the texture image, and the patch region of the depth image is based on facial landmarks defined in the personalized 3D face model. ### Claim 18 (depends on 14) The method of claim 14, wherein the affine transformation represents a transformation between the patch region of the depth image and the patch region of the texture image.