# US 10,796,480 B2 · Methods of generating personalized 3D head models or 3D body models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,796,480 B2 - Document ID: US10796480B2 - Status: Granted - Kind: B2 - Application number: 15/752,709 - Filed: 2016-08-15 - Published: 2019-01-31 - Granted: 2020-10-06 - Earliest claimed: 2015-08-14 - Assignees: Metail Limited - Applicants: Metail Limited - Inventors: Yu Chen; Dongjoe Shin; Joe Townsend; Jim Downing; Duncan Robertson; Tom Adeyoola - Web page: https://patentgrep.com/patent/US10796480B2 ## CPC - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 7/50: Depth or shape recovery - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 15/04: Texture mapping - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06V 40/165: {using facial parts and geometric relationships} - G06V 40/166: {using acquisition arrangements} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 2200/08: involving all processing steps from image acquisition to 3D model generation - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G06T 2215/16: Using real world measurements to influence rendering - G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes - G06T 2219/2021: Shape modification - H04N 23/64: {Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract There is provided a method of generating an image file of a personalized 3D head model of a user, the method comprising the steps of: (i) acquiring at least one 2D image of the user's face; (ii) performing automated face 2D landmark recognition based on the at least one 2D image of the user's face; (iii) providing a 3D face geometry reconstruction using a shape prior; (iv) providing texture map generation and interpolation with respect to the 3D face geometry reconstruction to generate a personalized 3D head model of the user, and (v) generating an image file of the personalized 3D head model of the user. A related system and computer program product are also provided. ## Claims ### Claim 1 A method of generating an image file of a personalized three-dimensional (3D) head model of a user, the method comprising the steps of: (i) acquiring at least one two-dimensional (2D) image of the user's face; (ii) performing automated face 2D landmark recognition based on the at least one 2D image of the user's face; (iii) providing a 3D face geometry reconstruction using a shape prior; (iv) providing texture map generation and interpolation with respect to the 3D face geometry reconstruction to generate the personalized 3D head model of the user, and (v) generating the image file of the personalized 3D head model of the user; wherein the generated texture map includes a 3D mesh geometry of the user's face comprising mesh triangles; UV texture coordinates are determined for texture vertices of each mesh triangle of the 3D mesh geometry of the user's face; in which a UV coordinate of a landmark vertex is computed based on a result of a corresponding 2D face landmark position detected by a 2D face landmark detector on the at least one 2D image of the user's face; in which head geometry is improved for better realism by deforming an initial head model by rectifying facial landmark positions of the personalized 3D head model in directions within an image plane of the at least one 2D image of the user's face, so that a projection of facial landmarks on the personalized 3D head model is a similarity transform of corresponding 2D facial landmarks in the at least one 2D image of the user's face. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one 2D image of the user's face is acquired via a network communication. ### Claim 3 (depends on 2) The method of claim 2, wherein the at least one 2D image of the user's face is acquired via the network communication, from a smartphone including a camera. ### Claim 4 (depends on 1) The method of claim 1, wherein the at least one 2D image of the user's face is a front image of the user's face. ### Claim 5 (depends on 1) The method of claim 1, wherein the at least one 2D image of the user's face is a smartphone camera image of the user's face. ### Claim 6 (depends on 1) The method of claim 1, wherein the automated face 2D landmark recognition includes using the 2D face landmark detector. ### Claim 7 (depends on 1) The method of claim 1, in which the image file is a 3D image file. ### Claim 8 (depends on 1) The method of claim 1, in which the image file is a 2D image file. ### Claim 9 (depends on 1) The method of claim 1, in which the image file is an animation file. ### Claim 10 (depends on 1) The method of claim 1, in which the image file is a personalised sticker set. ### Claim 11 (depends on 1) The method of claim 1, in which head size is estimated from body shape parameters. ### Claim 12 (depends on 1) The method of claim 1, in which an automatic image analysis is performed to help the user quickly acquire input data of good quality. ### Claim 13 (depends on 12) The method of claim 12, in which prior to starting the video or image capture, the user is presented with a live view of a camera feed, and a feedback mechanism analyses the live view and provides the user with recommendations on how to improve conditions in order to achieve a high quality end result. ### Claim 14 (depends on 1) The method of claim 1, in which a machine-learning-based attribute classifier, which can be implemented by a deep convolutional neural network (CNN), is used to analyze the at least one 2D image of the user's face, and predict attributes from appearance information in the at least one 2D image of the user's face. ### Claim 15 (depends on 1) The method of claim 1, in which a 3D thin-plate spline (TPS) deformation model is used to rectify a 3D geometry of a regressed head model to achieve better geometric similarity, so as to generate a smooth interpolation of 3D geometry deformation throughout the 3D mesh geometry of the user's face from control point pairs. ### Claim 16 (depends on 1) The method of claim 1, in which to complete the generated texture map of the personalized 3D head model, a 2D thin plate spline (TPS) model is used for interpolation and to populate the UV texture coordinates over the texture vertices of each mesh triangle of the 3D mesh geometry of the user's face. ### Claim 17 (depends on 1) The method of claim 1, in which the at least one 2D image of the user's face comprises at least a front image, a left side image and a right side image, of the user's face, and in which following generating an approximate 3D face model from the front image for use as an initialisation model, a step is performed of performing an iterative optimisation algorithm for revising the initialisation model, which is implemented to minimise landmark re-projection errors against independent 2D face landmark detection results obtained from the front image, the left side image and the right side image, of the user's face. ### Claim 18 A system including a processor, wherein the processor is configured to generate an image file of a personalized three-dimensional (3D) head model of a user, the processor configured to: (i) acquire at least one two-dimensional (2D) image of the user's face; (ii) perform automated face 2D landmark recognition based on the at least one 2D image of the user's face; (iii) provide a 3D face geometry reconstruction using a shape prior; (iv) provide texture map generation and interpolation with respect to the 3D face geometry reconstruction to generate the personalized 3D head model of the user, and (v) generate the image file of the personalized 3D head model of the user; wherein the generated texture map includes a 3D mesh geometry of the user's face comprising mesh triangles; UV texture coordinates are determined for texture vertices of each mesh triangle of the 3D mesh geometry of the user's face; in which a UV coordinate of a landmark vertex is computed based on a result of a corresponding 2D face landmark position detected by a 2D face landmark detector on the at least one 2D image of the user's face; in which head geometry is improved for better realism by deforming an initial head model by rectifying facial landmark positions of the personalized 3D head model in directions within an image plane of the at least one 2D image of the user's face, so that a projection of facial landmarks on the personalized 3D head model is a similarity transform of corresponding 2D facial landmarks in the at least one 2D image of the user's face. ### Claim 19 A computer program product embodied on a non-transitory storage medium, the computer program product comprising instructions executable on a processor to generate an image file of a personalized three-dimensional (3D) head model of a user, the computer program product executable on the processor to: (i) receive at least one two-dimensional (2D) image of the user's face; (ii) perform an automated face 2D landmark recognition based on the at least one 2D image of the user's face; (iii) provide a 3D face geometry reconstruction using a shape prior; (iv) provide texture map generation and interpolation with respect to the 3D face geometry reconstruction to generate the personalized 3D head model of the user, and (v) generate the image file of the personalized 3D head model of the user; wherein the generated texture map includes a 3D mesh geometry of the user's face comprising mesh triangles; UV texture coordinates are determined for texture vertices of each mesh triangle of the 3D mesh geometry of the user's face; in which a UV coordinate of a landmark vertex is computed based on a result of a corresponding 2D face landmark position detected by a 2D face landmark detector on the at least one 2D image of the user's face; in which head geometry is improved for better realism by deforming an initial head model by rectifying facial landmark positions of the personalized 3D head model in directions within an image plane of the at least one 2D image of the user's face, so that a projection of facial landmarks on the personalized 3D head model is a similarity transform of corresponding 2D facial landmarks in the at least one 2D image of the user's face. ### Claim 20 A method of generating an image file of a personalized three-dimensional (3D) head model of a user, the method comprising the steps of: (i) acquiring at least one two-dimensional (2D) image of the user's face; (ii) performing automated face 2D landmark recognition based on the at least one 2D image of the user's face; (iii) providing a 3D face geometry reconstruction using a shape prior; (iv) providing texture map generation and interpolation with respect to the 3D face geometry reconstruction to generate the personalized 3D head model of the user, and (v) generating the image file of the personalized 3D head model of the user; wherein the generated texture map includes a 3D mesh geometry of the user's face comprising mesh triangles; UV texture coordinates are determined for texture vertices of each mesh triangle of the 3D mesh geometry of the user's face; in which a UV coordinate of a landmark vertex is computed based on a result of a corresponding 2D face landmark position detected by a 2D face landmark detector on the at least one 2D image of the user's face; in which a 3D thin-plate spline (TPS) deformation model is used to rectify a 3D geometry of a regressed head model to achieve better geometric similarity, so as to generate a smooth interpolation of 3D geometry deformation throughout the 3D mesh geometry of the user's face from control point pairs. ### Claim 21 A method of generating an image file of a personalized three-dimensional (3D) head model of a user, the method comprising the steps of: (i) acquiring at least one two-dimensional (2D) image of the user's face; (ii) performing automated face 2D landmark recognition based on the at least one 2D image of the user's face; (iii) providing a 3D face geometry reconstruction using a shape prior; (iv) providing texture map generation and interpolation with respect to the 3D face geometry reconstruction to generate the personalized 3D head model of the user, and (v) generating the image file of the personalized 3D head model of the user; wherein the generated texture map includes a 3D mesh geometry of the user's face comprising mesh triangles; UV texture coordinates are determined for texture vertices of each mesh triangle of the 3D mesh geometry of the user's face; in which a UV coordinate of a landmark vertex is computed based on a result of a corresponding 2D face landmark position detected by a 2D face landmark detector on the at least one 2D image of the user's face; in which the at least one 2D image of the user's face comprises at least a front image, a left side image and a right side image, of the user's face, and in which following generating an approximate 3D face model from the front image for use as an initialisation model, a step is performed of performing an iterative optimisation algorithm for revising the initialisation model, which is implemented to minimise landmark re-projection errors against independent 2D face landmark detection results obtained from the front image, the left side image and the right side image, of the user's face.