Systems and methods for machine-generated avatars
AssigneeTrue Image Interactive, Inc.
InventorWayne Scholar
Life of this patent
Estimating the term…
Abstract
Systems and methods are disclosed for creating a machine generated avatar. A machine generated avatar is an avatar generated by processing video and audio information extracted from a recording of a human speaking a reading corpora and enabling the created avatar to be able to say an unlimited number of utterances, i.e., utterances that were not recorded. The video and audio processing consists of the use of machine learning algorithms that may create predictive models based upon pixel, semantic, phonetic, intonation, and wavelets.
Claims20
A method for implementing a machine-generated avatar, the method comprising:
- receiving data indicative of an input utterance;
- generating one or more phoneme sequences based on the data indicative of the input utterance;
- generating phoneme matrices comprising values based on one or more corresponding phonemes of the one or more phoneme sequences;
- matching, based on phoneme and duration, corresponding viseme clips to the phoneme matrices;
- generating transition matrices for phoneme pairs using a trained audio linear regression model;
- generating transition matrices for viseme frames based on a trained light cone linear regression model;
- modulating the phoneme matrices and the transition matrices for phoneme pairs to generate correct intonation using a trained modulation model;
- generating an audio track based on the modulated matrices;
- combining the audio track with visual frames;
- and
- causing storage of the combined audio track and visual frames.
A device for implementing a machine-generated avatar, the device comprising:
- one or more processors;
- and
- memory storing instructions that, when executed by the one or more processors, cause the device to:
- receive data indicative of an input utterance;
- generate one or more phoneme sequences based on the data indicative of the input utterance;
- generate phoneme matrices comprising values based on one or more corresponding phonemes of the one or more phoneme sequences;
- match, based on phoneme and duration, corresponding viseme clips to the phoneme matrices;
- generate transition matrices for phoneme pairs using a trained audio linear regression model;
- generate transition matrices for viseme frames based on a trained light cone linear regression model;
- modulate the phoneme matrices and the transition matrices for phoneme pairs to generate correct intonation using a trained modulation model;
- generate an audio track based on the modulated matrices;
- combine the audio track with visual frames;
- and
- cause storage of the combined audio track and visual frames.
A system for implementing a machine-generated avatar, the system comprising:
- a storage device;
- and
- a computing device configured to:
- receive data indicative of an input utterance;
- generate one or more phoneme sequences based on the data indicative of the input utterance;
- generate phoneme matrices comprising values based on one or more corresponding phonemes of the one or more phoneme sequences;
- match, based on phoneme and duration, corresponding viseme clips to the phoneme matrices;
- generate transition matrices for phoneme pairs using a trained audio linear regression model;
- generate transition matrices for viseme frames based on a trained light cone linear regression model;
- modulate the phoneme matrices and the transition matrices for phoneme pairs to generate correct intonation using a trained modulation model;
- generate an audio track based on the modulated matrices;
- combine the audio track with visual frames;
- and
- cause the storage device to store the combined audio track and visual frames.
Dependencies are inferred from the claim text.