Back to results
US 12,182,919 B2Granted
Joint audio-video facial animation system
AssigneeSnap Inc.
InventorsChen Cao, Xin Chen, Wei Chu, Zehao Xue
Life of this patent
Estimating the term…
Abstract
The present invention relates to a joint automatic audio visual driven facial animation system that in some example embodiments includes a full scale state of the art Large Vocabulary Continuous Speech Recognition (LVCSR) with a strong language model for speech recognition and obtained phoneme alignment from the word lattice.
Claims20
A method comprising:
- accessing a video data at a client device, the video data comprising a set of facial landmarks;
- identifying a user profile that corresponds with the set of facial landmarks;
- generating a facial model based on the user profile and the set of facial landmarks;
- causing display of a presentation of the facial model;
- receiving audio data that comprises a speech signal;
- determining a phone sequence based on the speech signal;
- animating the presentation of the facial model based on the phone sequence;
- detecting a loss in the audio data;
- and
- causing display of the presentation of the facial model based on the video data responsive to the loss in the audio data.
7 more elements
A system comprising:
- a memory;
- and
- at least one hardware processor coupled to the memory and comprising instructions that causes the system to perform operations comprising:
- accessing a video data at a client device, the video data comprising a set of facial landmarks;
- identifying a user profile that corresponds with the set of facial landmarks;
- generating a facial model based on the user profile and the set of facial landmarks;
- causing display of a presentation of the facial model;
- receiving audio data that comprises a speech signal;
- determining a phone sequence based on the speech signal;
- animating the presentation of the facial model based on the phone sequence;
- detecting a loss in the audio data;
- and
- causing display of the presentation of the facial model based on the video data responsive to the loss in the audio data.
10 more elements
A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:
- accessing a video data at a client device, the video data comprising a set of facial landmarks;
- identifying a user profile that corresponds with the set of facial landmarks;
- generating a facial model based on the user profile and the set of facial landmarks;
- causing display of a presentation of the facial model;
- receiving audio data that comprises a speech signal;
- determining a phone sequence based on the speech signal;
- animating the presentation of the facial model based on the phone sequence;
- detecting a loss in the audio data;
- and
- causing display of the presentation of the facial model based on the video data responsive to the loss in the audio data.
7 more elements
Dependencies are inferred from the claim text.