# US 2021/0279938 A1 · Appearance synthesis of digital faces > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279938 A1 - Document ID: US20210279938A1 - Status: Published application - Kind: A1 - Application number: 16/895,734 - Filed: 2020-06-08 - Published: 2021-09-09 - Earliest claimed: 2020-03-05 - Assignees: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich) - Applicants: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich) - Inventors: Prashanth CHANDRAN; Dominik Thabo BEELER; Derek Edward BRADLEY - Web page: https://patentgrep.com/patent/US20210279938A1 ## CPC - G06T 15/04: Texture mapping - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06T 15/10: Geometric effects - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics - G06T 2200/04: involving 3D image data - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Techniques are disclosed for generating digital faces. In some examples, a style-based generator receives as inputs initial tensor(s) and style vector(s) corresponding to user-selected semantic attribute styles, such as the desired expression, gender, age, identity, and/or ethnicity of a digital face. The style-based generator is trained to process such inputs and output low-resolution appearance map(s) for the digital face, such as a texture map, a normal map, and/or a specular roughness map. The low-resolution appearance map(s) are further processed using a super-resolution generator that is trained to take the low-resolution appearance map(s) and low-resolution 3D geometry of the digital face as inputs and output high-resolution appearance map(s) that align with high-resolution 3D geometry of the digital face. Such high-resolution appearance map(s) and high-resolution 3D geometry can then be used to render standalone images or the frames of a video that include the digital face. ## Claims ### Claim 1 A computer-implemented method for rendering one or more images of a digital face, the method comprising: generating, via a first machine learning model, one or more first appearance maps based on a user selection of one or more styles associated with one or more attributes of a digital face; generating, via a second machine learning model, one or more second appearance maps and a first three-dimensional (3D) geometry associated with the digital face based on the one or more first appearance maps and a second 3D geometry associated with the digital face; and rendering one or more images including the digital face based on the one or more second appearance maps and the first 3D geometry. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the one or more first appearance maps have a lower resolution than the one or more second appearance maps, and the second 3D geometry has a lower resolution than the first 3D geometry. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein generating the one or more first appearance maps based on the user selection of one or more styles comprises controlling one or more adaptive instance normalization (AdaIN) operations based on the user selection of one or more styles. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the one or more AdaIN operations are performed in conjunction with one or more convolution operations. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the first machine learning model comprises a plurality of semantics transfer blocks, and performing the one or more AdaIN operations in conjunction with the one or more convolution operations comprises, for each semantics transfer block included in the plurality of semantics transfer blocks: performing multiple sets of convolution operations and AdaIN operations in parallel; determining a weighted sum of outputs of the multiple sets of convolution operations and AdaIN operations; and upscaling the weighted sum of outputs. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the one or more second appearance maps include at least one of a texture map, a normal map, or a specular roughness map. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the one or more attributes of the digital face include at least one of expression, gender, age, identity, or ethnicity. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the second machine learning model comprises a super-resolution generator. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the first 3D geometry associated with the digital face conveys a non-neutral facial expression. ### Claim 10 A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to perform steps for rendering one or more images of a digital face, the steps comprising: generating, via a first machine learning model, one or more first appearance maps based on a user selection of one or more styles associated with one or more attributes of a digital face; generating, via a second machine learning model, one or more second appearance maps and a first three-dimensional (3D) geometry associated with the digital face based on the one or more first appearance maps and a second 3D geometry associated with the digital face; and rendering one or more images including the digital face based on the one or more second appearance maps and the first 3D geometry. ### Claim 11 (depends on 10) The computer-readable storage medium of claim 10, wherein generating the one or more first appearance maps based on the user selection of one or more styles comprises controlling one or more adaptive instance normalization (AdaIN) operations based on the user selection of one or more styles. ### Claim 12 (depends on 11) The computer-readable storage medium of claim 11, wherein the first machine learning model comprises a plurality of semantics transfer blocks, and generating the one or more first appearance maps based on the user selection of one or more styles includes, for each semantics transfer block included in the plurality of semantics transfer blocks: performing multiple sets of convolution operations and AdaIN operations in parallel; determining a weighted sum of outputs of the multiple sets of convolution operations and AdaIN operations; and upscaling the weighted sum of outputs. ### Claim 13 (depends on 12) The computer-readable storage medium of claim 12, wherein one or more weights used in the weighted sum of outputs, the multiple sets of convolution operations and AdaIN operations, one or more initial tensors, and one or more style vectors associated with the one or more attributes of the digital face are determined while training the first machine learning model. ### Claim 14 (depends on 10) The computer-readable storage medium of claim 10, wherein the first machine learning model is trained using a progressive training technique. ### Claim 15 (depends on 10) The computer-readable storage medium of claim 10, wherein the second machine learning model is trained using ground truth and adversarial learning techniques. ### Claim 16 (depends on 10) The computer-readable storage medium of claim 10, wherein the one or more first appearance maps have a lower resolution than the one or more second appearance maps, and the second 3D geometry has a lower resolution than the first 3D geometry. ### Claim 17 (depends on 10) The computer-readable storage medium of claim 10, wherein the one or more second appearance maps include at least one of a texture map, a normal map, or a specular roughness map. ### Claim 18 (depends on 10) The computer-readable storage medium of claim 10, wherein the one or more attributes of the digital face include at least one of expression, gender, age, identity, or ethnicity. ### Claim 19 (depends on 10) The computer-readable storage medium of claim 10, wherein the second machine learning model comprises a super-resolution generator. ### Claim 20 A computing device comprising: a memory storing an application; and a processor coupled to the memory, wherein when executed by the processor, the application causes the processor to: generate, via a first machine learning model, one or more first appearance maps based on a user selection of one or more styles associated with one or more attributes of a digital face; generate, via a second machine learning model, one or more second appearance maps and a first three-dimensional (3D) geometry associated with the digital face based on the one or more first appearance maps and a second 3D geometry associated with the digital face; and render one or more images including the digital face based on the one or more second appearance maps and the first 3D geometry.