# US 11,308,657 B1 · Methods and systems for image processing using a learning engine > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,657 B1 - Document ID: US11308657B1 - Status: Granted - Kind: B1 - Application number: 17/399,962 - Filed: 2021-08-11 - Granted: 2022-04-19 - Earliest claimed: 2021-08-11 - Assignees: Neon Evolution Inc. - Applicants: Neon Evolution Inc. - Inventors: Cody Gustave Berlin; Carl Davis Bogan, III; Kenneth Michael Lande; Anders Øland; Davide Toniolo; Alessia Bertugli; Dario Bertazioli; Brian Sung Lee - Web page: https://patentgrep.com/patent/US11308657B1 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G06T 11/00: Two-dimensional [2D] image generation - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06N 3/096: Transfer learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/60: using machine learning, e.g. neural networks - G06T 5/77: Retouching; Inpainting; Scratch removal - G06V 40/174: {Facial expression recognition} - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30196: Human being; Person - G06T 2207/30201: Face - G10L 21/013: Adapting to target pitch - G10L 25/18: the extracted parameters being spectral information of each sub-band - G10L 25/30: using neural networks - G10L 2021/0135: {Voice conversion or morphing} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Systems and methods are disclosed configured to train an autoencoder. A data training set is generated comprising images of different faces. A first autoencoder configuration is generated, comprising a first encoder, and a first decoder. The first autoencoder configuration is trained using dataset images, wherein weights associated with the first encoder and weights associated with the first decoder are modified. A second autoencoder configuration is generated comprising the first encoder and a second decoder. The second decoder is trained using a plurality of images of a first target face. First encoder weights are substantially maintained, and weights associated with the second decoder are modified. An autoencoder comprising the trained first encoder and the trained second decoder is used to generate an output using a source image of a first face having a facial expression, where the facial expression of the first face from the source image is applied to the first specific target face. ## Claims ### Claim 1 An electronic image processing system, comprising: a network interface; at least one computing device; computer readable memory including instructions operable to be executed by the at least one computing device to perform a set of actions, configuring the at least one computing device to perform operations comprising: access at least one hundred thousand captured images, comprising images of different faces; access a first autoencoder configuration, comprising: a first encoder, and a first decoder; train, using at least a pixelwise evaluator, the first autoencoder configuration comprising the first encoder and the first decoder using the at least one hundred thousand captured images, wherein node weights associated with the first encoder and node weights associated with the first decoder are modified in response to an output of the pixelwise evaluator; access a second autoencoder configuration, comprising the first encoder, trained in the first autoencoder configuration, and a second decoder; train with respect to a first specific target face, using a plurality of images of the first specific target face and at least adversarial criteria, the second decoder in the second autoencoder configuration, wherein node weights associated with the first encoder are fixed and not modified, wherein node weights associated with the second decoder are modified; use an autoencoder configuration comprising: the first encoder, trained in first autoencoder configuration using the at least one hundred thousand captured images, comprising images of different face, and the second decoder, trained using the plurality of images of the first specific target face, to generate an output using a source image of a first face having a facial expression, where the facial expression of the first face from the source image is applied to the first specific target face. ### Claim 2 (depends on 1) The electronic image processing system as defined in claim 1, wherein the training of the second decoder with respect to a first specific target face, using a plurality of images of the first specific target face and at least adversarial criteria further comprises using a patch-based discriminator in training the second decoder. ### Claim 3 (depends on 1) The electronic image processing system as defined in claim 1, wherein the training of the second decoder with respect to a first specific target face, using a plurality of images of the first specific target face and at least adversarial criteria further comprises using a multi-scale adversarial loss. ### Claim 4 (depends on 1) The electronic image processing system as defined in claim 1, wherein the training, using at least a pixelwise evaluator, the first autoencoder configuration comprising the first encoder and the first decoder further comprises using a sum of squared errors (SSE) or mean squared error (MSE) function. ### Claim 5 (depends on 1) The electronic image processing system as defined in claim 1, wherein the first specific target face comprises a CGI face. ### Claim 6 (depends on 1) The electronic image processing system as defined in claim 1, wherein the first specific target face comprises an actual human face. ### Claim 7 (depends on 1) The electronic image processing system as defined in claim 1, the operations further comprising: accessing a plurality of images of a given face; determining if the plurality of images of the given face satisfy at least a first temporal consistency criterion; and at least partly in response to determining that the plurality of images of the given face satisfy at least the first temporal consistency criterion, include the given image in a training dataset of images, configured to be used to train at least an autoencoder encoder. ### Claim 8 (depends on 1) The electronic image processing system as defined in claim 1, wherein the autoencoder configuration, comprising the trained first encoder and the trained second decoder, includes one or more intermediate networks. ### Claim 9 (depends on 1) The electronic image processing system as defined in claim 1, wherein the autoencoder, comprising the trained first encoder and the trained second decoder, has a HD, 4K UHD, 4K, 8K, or 16K output resolution. ### Claim 10 A computer-implemented method comprising: under control of a hardware computing device configured with specific computer-executable instructions: accessing a data training set comprising images of different faces; accessing a first autoencoder configuration, comprising: a first encoder, and a first decoder; training the first autoencoder configuration comprising the first encoder and the first decoder using dataset images, wherein node weights associated with the first encoder and node weights associated with the first decoder are modified during the training of the first autoencoder configuration; accessing a second autoencoder configuration, comprising: the first encoder, and a second decoder; training with respect to a first target face, using a plurality of images of the first target face, the second decoder in the second autoencoder configuration, wherein node weights associated with the first encoder are fixed, wherein node weights associated with the second decoder are modified; using an autoencoder comprising: the first encoder, trained using the using dataset images, and the trained second decoder, trained using the plurality of images of the first target face, to generate an output using a source image of a first face having a facial expression, where the facial expression of the first face from the source image is applied to the first specific target face. ### Claim 11 (depends on 10) The method as defined in claim 10, wherein the training of the second decoder with respect to a first target face, using a plurality of images of the first target face comprises using a patch-based discriminator in training the second decoder. ### Claim 12 (depends on 10) The method as defined in claim 10, wherein the training of the second decoder with respect to a first target face, using a plurality of images of the first target face further comprises using a multi-scale adversarial loss. ### Claim 13 (depends on 10) The method as defined in claim 10, wherein the training the first autoencoder configuration comprising the first encoder and the first decoder further comprises, using at least a pixelwise evaluator. ### Claim 14 (depends on 10) The method as defined in claim 10, wherein the first target face comprises a CGI face. ### Claim 15 (depends on 10) The method as defined in claim 10, wherein the first target face comprises an actual human face. ### Claim 16 (depends on 10) The method as defined in claim 10, the method further comprising: accessing a plurality of images of a given face; determining if the plurality of images of the given face satisfy at least a first temporal consistency criterion; and at least partly in response to determining that the plurality of images of the given face satisfy at least the first temporal consistency criterion, include the given image in a training dataset of images, configured to be used to train at least an autoencoder encoder. ### Claim 17 (depends on 10) The method as defined in claim 10, wherein the trained autoencoder, comprising the trained first encoder and the trained second decoder, includes one or more intermediate networks. ### Claim 18 (depends on 10) The method as defined in claim 10, wherein the trained autoencoder, comprising the trained first encoder and the trained second decoder, has a HD, 4K UHD, 4K, 8K, or 16K output resolution. ### Claim 19 An electronic image processing system, comprising: at least one computing device; computer readable memory including instructions operable to be executed by the at least one computing device to perform a set of actions, configuring the at least one computing device to perform operations comprising: access a data training set comprising images of different faces; access a first autoencoder configuration, comprising: a first encoder, and a first decoder; train the first autoencoder configuration comprising the first encoder and the first decoder using dataset images, wherein node weights associated with the first encoder and node weights associated with the first decoder are modified during the training of the first autoencoder configuration; access a second autoencoder configuration, comprising: the first encoder, and a second decoder; train with respect to a first target face, using a plurality of images of the first target face, the second decoder in the second autoencoder configuration, wherein node weights associated with the first encoder are substantially maintained, wherein node weights associated with the second decoder are modified; use an autoencoder comprising: the first encoder, trained using the dataset images, and the second decoder, trained using the plurality of images of the first target face, to generate an output using a source image of a first face having a facial expression, where the facial expression of the first face from the source image is applied to the first specific target face. ### Claim 20 (depends on 19) The electronic image processing system as defined in claim 19, wherein the training of the second decoder with respect to a first target face, using a plurality of images of the first target face, comprises using a patch-based discriminator in training the second decoder. ### Claim 21 (depends on 19) The electronic image processing system as defined in claim 19, wherein the training of the second decoder with respect to a first target face, using a plurality of images of the first target face, further comprises using a multi-scale adversarial loss. ### Claim 22 (depends on 19) The electronic image processing system as defined in claim 19, wherein the training of the first autoencoder configuration comprising the first encoder and the first decoder further comprises, using at least a pixelwise evaluator. ### Claim 23 (depends on 19) The electronic image processing system as defined in claim 19, wherein the first target face comprises a CGI face. ### Claim 24 (depends on 19) The electronic image processing system as defined in claim 19, wherein the first target face comprises an actual human face.