# US 11,114,086 B2 · Text and audio-based real-time face reenactment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,114,086 B2 - Document ID: US11114086B2 - Status: Granted - Kind: B2 - Application number: 16/509,370 - Filed: 2019-07-11 - Published: 2020-07-23 - Granted: 2021-09-07 - Earliest claimed: 2019-01-18 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Pavel Savchenkov; Maxim Lukin; Aleksandr Mashrabov - Web page: https://patentgrep.com/patent/US11114086B2 ## CPC - G10L 13/00: Speech synthesis; Text to speech systems - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Provided are systems and methods for text and audio-based real-time face reenactment. An example method includes receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; determining, based on the sequence of sets of acoustic features, a sequence of sets of scenario data indicating modifications of the target face for pronouncing the input text; generating, based on the sequence of sets of scenario data, a sequence of frames, wherein each of the frames includes the target face modified based on at least one of the sets of scenario data; generating, based on the sequence of frames, an output video; and synthesizing, based on the sequence of sets of acoustic features, an audio data and adding the audio data to the output video. ## Claims ### Claim 1 A method for text and audio-based real-time face reenactment, the method comprising: receiving, by a computing device, an input text and a target image, the target image including a target face; generating, by the computing device and based on the input text, a sequence of sets of acoustic features representing the input text; generating, by the computing device and based on the sequence of sets of acoustic features, a sequence of sets of scenario data, the sets of scenario data indicating modifications of facial key points of a model face pronouncing the input text, wherein the generating the sequence of sets of scenario data includes: generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points of the model face by providing the sets of acoustic features to a first model, the first model being trained to generate, based on a set of acoustic features, a set of mouth key points, the first model being trained based on first images of a first actor pronouncing first training input texts; and generating, based on the sequence of sets of mouth key points, a sequence of sets of the facial key points of the model face by providing the sets of mouth key points to a second model, the second model being trained to generate, based on the set of mouth key points, a set of the facial key points, the second model being trained based on second images of a second actor pronouncing second training input texts; generating, by the computing device and based on the sequence of sets of scenario data and the target image, a sequence of frames, wherein each of the frames includes the target face modified based on at least one set of scenario data of the sequence of sets of scenario data; and generating, by the computing device and based on the sequence of frames, an output video. ### Claim 2 (depends on 1) The method of claim 1, further comprising: synthesizing, by the computing device and based on the sequence of sets of acoustic features, an audio data representing the input text; and adding, by the computing device, the audio data to the output video. ### Claim 3 (depends on 1) The method of claim 1, wherein the acoustic features include Mel- frequency cepstral coefficients. ### Claim 4 (depends on 1) The method of claim 1, wherein the sequence of sets of acoustic features is generated by a neural network. ### Claim 5 (depends on 1) The method of claim 1, wherein: the generating the sequence of frames includes: determining, based on a sequence of sets of the facial key points, a sequence of sets of two-dimensional (2D) deformations; and applying each set of 2D deformations of the sequence of the sets of 2D deformations to the target image to obtain the sequence of frames. ### Claim 6 (depends on 5) The method of claim 5, wherein: the sequence of sets of mouth key points is generated by a neural network; and at least one set of the sequence of sets of mouth key points is generated based on a pre-determined number of sets preceding the at least one set in the sequence of sets of mouth key points. ### Claim 7 (depends on 6) The method of claim 6, wherein: the at the least one set of the sequence of sets of mouth key points corresponds to at least one set (S) of the sequence of sets of acoustic features; and the at least one set of the sequence of sets of mouth key points is generated based on a first pre-determined number of sets of acoustic features preceding the S in the sequence of sets of acoustic features and a second pre-determined number sets of acoustic features succeeding the S in the sequence of sets of acoustic features. ### Claim 8 (depends on 5) The method of claim 5, wherein: the sequence of sets of facial key points is generated by a neural network; and at least one set of the sequence of sets of facial key points is determined based on a pre-determined number of sets preceding the at least one set in the sequence of sets of facial key points. ### Claim 9 (depends on 5) The method of claim 5, further comprising: generating, by the computing device and based on the sequence of sets of mouth key points, a sequence of mouth texture images; and inserting, by the computing device, each of the sequence of mouth texture images in a corresponding frame of the sequence of the frames. ### Claim 10 (depends on 9) The method of claim 9, wherein each mouth texture image of the sequence of mouth texture images is generated by a neural network based on a first pre-determined number of mouth texture images preceding the mouth region image in the sequence of mouth region images. ### Claim 11 A system for text and audio-based real-time face reenactment, the system comprising at least one processor, a memory storing processor-executable codes, wherein the at least one processor is configured to implement the following operations upon executing the processor-executable codes: receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; generating, based on the sequence of sets of acoustic features, a sequence of sets of scenario data, the sets of scenario data indicating modifications of facial key points of a model face pronouncing the input text, wherein the generating the sequence of sets of scenario data includes: generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points of the model face by providing the sets of acoustic features to a first model, the first model being trained to generate, based on a set of acoustic features, a set of mouth key points, the first model being trained based on first images of a first actor pronouncing first training input texts; and generating, based on the sequence of sets of mouth key points, a sequence of sets of the facial key points of the model face by providing the sets of mouth key points to a second model, the second model being trained to generate, based on the set of mouth key points, a set of the facial key points, the second model being trained based on second images of a second actor pronouncing second training input texts; generating, based on the sequence of sets of scenario data and the target image, a sequence of frames, wherein each of the frames includes the target face modified based on at least one set of scenario data of the sequence of sets of scenario data; and generating, based on the sequence of frames, an output video. ### Claim 12 (depends on 11) The system of claim 11, further comprising: synthesizing, based on the sequence of sets of acoustic features, an audio data representing the input text; and adding the audio data to the output video. ### Claim 13 (depends on 11) The system of claim 11, wherein the acoustic features include Mel-frequency cepstral coefficients. ### Claim 14 (depends on 11) The system of claim 11, wherein the sequence of sets of acoustic features is generated based on a neural network. ### Claim 15 (depends on 11) The system of claim 11, wherein: the generating the sequence of frames includes: determining, based on a sequence of sets of the facial key points, a sequence of sets of 2D deformations; and applying each set of two-dimensional (2D) deformations of the sequence of the sets of 2D deformations to the target image to obtain the sequence of frames. ### Claim 16 (depends on 15) The system of claim 15, wherein: the sequence of sets of mouth key points is generated by a neural network; and at least one set of the sequence of sets of mouth key points is generated based on a pre-determined number of sets preceding the at least one set in the sequence of sets of mouth key points. ### Claim 17 (depends on 16) The system of claim 16, wherein: the at the least one set of the sequence of sets of mouth key points corresponds to at least one set (S) of the sequence of sets of acoustic features; and the at least one set of the sequence of sets of mouth key points is generated based on a first pre-determined number of sets of acoustic features preceding the S in the sequence of sets of acoustic features and a second pre-determined number sets of acoustic features succeeding the S in the sequence of sets of acoustic features. ### Claim 18 (depends on 15) The system of claim 15, wherein: the sequence of sets of facial key points is generated by a neural network; and at least one set of the sequence of sets of facial key points is generated based on a pre-determined number of sets preceding the at least one set in the sequence of sets of facial key points. ### Claim 19 (depends on 15) The system of claim 15, further comprising: generating, by the computing device and based on the sequence of sets of mouth key points, a sequence of mouth texture images; and inserting, by the computing device, each of the sequence of mouth texture images in a corresponding frame of the sequence of the frames. ### Claim 20 A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method for text and audio- based real-time face reenactment, the method comprising: receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; generating, based on the sequence of sets of acoustic features, a sequence of sets of scenario data, the sets of scenario data indicating modifications of facial key points of a model face pronouncing the input text, wherein the generating the sequence of sets of scenario data includes: generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points of the model face by providing the sets of acoustic features to a first model, the first model being trained to generate, based on a set of acoustic features, a set of mouth key points, the first model being trained based on first images of a first actor pronouncing first training input texts; and generating, based on the sequence of sets of mouth key points, a sequence of sets of the facial key points of the model face by providing the sets of mouth key points to a second model, the second model being trained to generate, based on the set of mouth key points, a set of the facial key points, the second model being trained based on second images of a second actor pronouncing second training input texts; generating, based on the sequence of sets of scenario data and the target image, a sequence of frames, wherein each of the frames includes the target face modified based on at least one set of scenario data of the sequence of sets of scenario data; and generating, based on the sequence of frames, an output video.