# US 12,182,919 B2 · Joint audio-video facial animation system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,919 B2 - Document ID: US12182919B2 - Status: Granted - Kind: B2 - Application number: 18/064,140 - Filed: 2022-12-09 - Published: 2023-04-06 - Granted: 2024-12-31 - Earliest claimed: 2017-10-26 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Chen Cao; Xin Chen; Wei Chu; Zehao Xue - Web page: https://patentgrep.com/patent/US12182919B2 ## CPC - G06T 13/205: {driven by audio data} - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06V 20/64: Three-dimensional [3D] objects - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/176: {Dynamic expression} - G10L 21/10: Transforming into visible information - H04L 51/08: Annexed information, e.g. attachments - H04L 51/10: Multimedia information - H04R 27/00: Public address systems (circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; {monitoring or testing arrangements for public address systems H04R29/007}; amplifiers H03F) - G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads} - H04L 51/222: using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract The present invention relates to a joint automatic audio visual driven facial animation system that in some example embodiments includes a full scale state of the art Large Vocabulary Continuous Speech Recognition (LVCSR) with a strong language model for speech recognition and obtained phoneme alignment from the word lattice. ## Claims ### Claim 1 A method comprising: accessing a video data at a client device, the video data comprising a set of facial landmarks; identifying a user profile that corresponds with the set of facial landmarks; generating a facial model based on the user profile and the set of facial landmarks; causing display of a presentation of the facial model; receiving audio data that comprises a speech signal; determining a phone sequence based on the speech signal; animating the presentation of the facial model based on the phone sequence; detecting a loss in the audio data; and causing display of the presentation of the facial model based on the video data responsive to the loss in the audio data. ### Claim 2 (depends on 1) The method of claim 1, wherein the facial model includes a three-dimensional (3D) facial model. ### Claim 3 (depends on 1) The method of claim 1, wherein the client device is a first client device, and the causing display of the presentation of the facial model includes: causing display of the presentation of the facial model at a second client device. ### Claim 4 (depends on 1) The method of claim 1, wherein the causing display of the presentation of the facial model includes: causing display of an ephemeral message that includes the presentation of the facial model. ### Claim 5 (depends on 1) The method of claim 1, wherein the determining the phone sequence based on the speech signal further comprises: accessing the trained acoustic model, the trained acoustic model comprising a recurrent neural network trained on a dataset of speech signals; and determining the phone sequence based on the trained acoustic model and the speech signal. ### Claim 6 (depends on 1) The method of claim 1, wherein the user profile comprises a selection of a user avatar, and wherein the generating the facial model includes: generating the facial model based on the selection of the user avatar and the set of facial landmarks. ### Claim 7 (depends on 1) The method of claim 1, wherein the receiving the audio data that comprises the speech signal further comprises: passing the audio data through a Mel frequency scanned filter bank; and determining the speech signal based on the Mel frequency scanned filter bank. ### Claim 8 A system comprising: a memory; and at least one hardware processor coupled to the memory and comprising instructions that causes the system to perform operations comprising: accessing a video data at a client device, the video data comprising a set of facial landmarks; identifying a user profile that corresponds with the set of facial landmarks; generating a facial model based on the user profile and the set of facial landmarks; causing display of a presentation of the facial model; receiving audio data that comprises a speech signal; determining a phone sequence based on the speech signal; animating the presentation of the facial model based on the phone sequence; detecting a loss in the audio data; and causing display of the presentation of the facial model based on the video data responsive to the loss in the audio data. ### Claim 9 (depends on 8) The system of claim 8, wherein the facial model includes a three-dimensional (3D) facial model. ### Claim 10 (depends on 8) The system of claim 8, wherein the client device is a first client device, and the causing display of the presentation of the facial model includes: causing display of the presentation of the facial model at a second client device. ### Claim 11 (depends on 8) The system of claim 8, wherein the causing display of the presentation of the facial model includes: causing display of an ephemeral message that includes the presentation of the facial model. ### Claim 12 (depends on 8) The system of claim 8, wherein the determining the phone sequence based on the speech signal further comprises: accessing the trained acoustic model, the trained acoustic model comprising a recurrent neural network trained on a dataset of speech signals; and determining the phone sequence based on the trained acoustic model and the speech signal. ### Claim 13 (depends on 8) The system of claim 8, wherein the user profile comprises a selection of a user avatar, and wherein the generating the facial model includes: generating the facial model based on the selection of the user avatar and the set of facial landmarks. ### Claim 14 (depends on 8) The system of claim 8, wherein the receiving the audio data that comprises the speech signal further comprises: passing the audio data through a Mel frequency scanned filter bank; and determining the speech signal based on the Mel frequency scanned filter bank. ### Claim 15 A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: accessing a video data at a client device, the video data comprising a set of facial landmarks; identifying a user profile that corresponds with the set of facial landmarks; generating a facial model based on the user profile and the set of facial landmarks; causing display of a presentation of the facial model; receiving audio data that comprises a speech signal; determining a phone sequence based on the speech signal; animating the presentation of the facial model based on the phone sequence; detecting a loss in the audio data; and causing display of the presentation of the facial model based on the video data responsive to the loss in the audio data. ### Claim 16 (depends on 15) The non-transitory machine-readable storage medium of claim 15, wherein the facial model includes a three-dimensional (3D) facial model. ### Claim 17 (depends on 15) The non-transitory machine-readable storage medium of claim 15, wherein the client device is a first client device, and the causing display of the presentation of the facial model includes: causing display of the presentation of the facial model at a second client device. ### Claim 18 (depends on 15) The non-transitory machine-readable storage medium of claim 15, wherein the causing display of the presentation of the facial model includes: causing display of an ephemeral message that includes the presentation of the facial model. ### Claim 19 (depends on 15) The non-transitory machine-readable storage medium of claim 15, wherein the determining the phone sequence based on the speech signal further comprises: accessing the trained acoustic model, the trained acoustic model comprising a recurrent neural network trained on a dataset of speech signals; and determining the phone sequence based on the trained acoustic model and the speech signal. ### Claim 20 (depends on 15) The non-transitory machine-readable storage medium of claim 15, wherein the user profile comprises a selection of a user avatar, and wherein the generating the facial model includes: generating the facial model based on the selection of the user avatar and the set of facial landmarks.