# US 2021/0279956 A1 · Semantic deep face models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279956 A1 - Document ID: US20210279956A1 - Status: Published application - Kind: A1 - Application number: 16/809,495 - Filed: 2020-03-04 - Published: 2021-09-09 - Earliest claimed: 2020-03-04 - Assignees: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich) - Applicants: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich) - Inventors: Prashanth CHANDRAN; Dominik Thabo BEELER; Derek Edward BRADLEY - Web page: https://patentgrep.com/patent/US20210279956A1 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06V 10/754: {involving a deformation of the sample pattern or of the reference pattern; Elastic matching} - G06V 40/168: {Feature extraction; Face representation} - G06V 40/175: {Static expression} - G06T 2210/44: Morphing - G06T 2219/2021: Shape modification - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques are disclosed for training and applying nonlinear face models. In embodiments, a nonlinear face model includes an identity encoder, an expression encoder, and a decoder. The identity encoder takes as input a representation of a facial identity, such as a neutral face mesh minus a reference mesh, and outputs a code associated with the facial identity. The expression encoder takes as input a representation of a target expression, such as a set of blendweight values, and outputs a code associated with the target expression. The codes associated with the facial identity and the facial expression can be concatenated and input into the decoder, which outputs a representation of a face having the facial identity and expression. The representation of the face can include vertex displacements for deforming the reference mesh. ## Claims ### Claim 1 A computer-implemented method for generating a representation of a face, the method comprising: receiving a representation of a facial identity and a representation of a facial expression; and performing one or more decoding operations based on an encoded representation of the facial identity and an encoded representation of the facial expression to generate the representation of the face having the facial identity and the facial expression. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the representation of the facial identity includes a mesh of the face having a neutral expression, and the method further comprises generating the encoded representation of the facial identity by: subtracting, from the mesh of the face, a reference mesh of a face to determine a set of differences between the mesh of the face and the reference mesh; and encoding the set of differences to generate the encoded representation of the facial identity. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the representation of the facial identity comprises the encoded representation of the facial identity. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising generating the encoded representation of the facial identity by adding noise to an encoded representation of another facial identity. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the representation of the facial expression includes one or more two-dimensional (2D) landmarks of facial features associated with an image depicting a face, and the method further comprises generating the encoded representation of the facial expression by mapping the one or more 2D landmarks to the encoded representation of the facial expression. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the representation of the facial expression includes one or more blendweight values, and the method further comprises performing one or more encoding operations based on the one or more blendweight values to generate the encoded representation of the facial expression. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, further comprising determining the one or more blendweight values based on one or more images. ### Claim 8 (depends on 6) The computer-implemented method of claim 6, further comprising determining the one or more blendweight values based on one or more three-dimensional meshes. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the representation of the facial expression comprises the encoded representation of the facial expression. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, further comprising rendering at least one image based on the representation of the face. ### Claim 11 A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to perform steps for generating a representation of a face, the steps comprising: receiving a representation of a facial identity and a representation of a facial expression; and processing an encoded representation of the facial identity and an encoded representation of the facial expression using a decoder to generate the representation of the face having the facial identity and the facial expression. ### Claim 12 (depends on 11) The computer-readable storage medium of claim 11, the steps further comprising: processing the representation of the facial identity using a first variational autoencoder (VAE) encoder to generate the encoded representation of the facial identity; and processing the representation of the facial expression using a second VAE encoder to generate the encoded representation of the facial expression. ### Claim 13 (depends on 11) The computer-readable storage medium of claim 11, wherein processing the encoded representation of the facial identity and the encoded representation of the facial expression using the decoder comprises: concatenating the encoded representation of the facial identity and the encoded representation of the facial expression to generate an encoded representation of the facial identity and the facial expression; and inputting the encoded representation of the facial identity and the facial expression into the decoder. ### Claim 14 (depends on 11) The computer-readable storage medium of claim 11, wherein the representation of the facial identity includes a mesh of the face having a neutral expression, and the steps further comprise generating the encoded representation of the facial identity by: subtracting, from the mesh of the face, a reference mesh of a face to determine a set of differences between the mesh of the face and the reference mesh; and encoding the set of differences to generate the encoded representation of the facial identity. ### Claim 15 (depends on 11) The computer-readable storage medium of claim 11, wherein the representation of the facial identity comprises the encoded representation of the facial identity, and the encoded representation of the facial identity is generated by adding noise to an encoded representation of another facial identity. ### Claim 16 (depends on 11) The computer-readable storage medium of claim 11, wherein the representation of the facial expression includes one or more two-dimensional (2D) landmarks of facial features associated with an image depicting a face, and the steps further comprise generating the encoded representation of the facial expression by mapping the one or more 2D landmarks to the encoded representation of the facial expression. ### Claim 17 (depends on 11) The computer-readable storage medium of claim 11, wherein the representation of the facial expression includes one or more blendweight values, and the steps further comprise performing one or more encoding operations based on the one or more blendweight values to generate the encoded representation of the facial expression. ### Claim 18 (depends on 17) The computer-readable storage medium of claim 17, the steps further comprising: receiving the one or more blendweight values via a user interface; or determining the one or more blendweight values based on at least one image or three-dimensional mesh. ### Claim 19 (depends on 11) The computer-readable storage medium of claim 11, the steps further comprising rendering at least one image based on the representation of the face. ### Claim 20 A computer-implemented method of training a model for generating a representation of a face, the method comprising: receiving a data set that includes meshes of neutral faces and meshes of faces having expressions; and training a model based on the data set, wherein the model comprises: a first encoder configured to map representations of neutral faces to encoded representations of facial identities, a second encoder configured to map representations of facial expressions to encoded representations of the facial expressions, and a decoder configured to map encoded representations of facial identities and encoded representations of facial expressions to representations of faces. ### Claim 21 (depends on 20) The computer-implemented method of claim 20, wherein training the model based on the data set comprises: subtracting each of the meshes of neutral faces from a reference mesh to determine a respective set of differences, and determining a respective set of blendweight values corresponding to a facial expression in each of the meshes of faces having expressions; and training the model based on the sets of differences and the sets of blendweight values.