Systems and methods for machine-generated avatars
AssigneeTrue Image Interactive, Inc.
InventorWayne SCHOLAR
Life of this patent
Estimating the term…
Abstract
Systems and methods are disclosed for creating a machine generated avatar. A machine generated avatar is an avatar generated by processing video and audio information extracted from a recording of a human speaking a reading corpora and enabling the created avatar to be able to say an unlimited number of utterances, i.e., utterances that were not recorded. The video and audio processing consists of the use of machine learning algorithms that may create predictive models based upon pixel, semantic, phonetic, intonation, and wavelets.
Claims2
A method for creating a machine-generated avatar, comprising:
- one or more reprocessing and training steps, comprising:
- receiving video and audio recording data;
- timestamping the recording data with phoneme times;
- extracting, based on the timestamping, phoneme clips and viseme clips;
- storing individual phoneme instances and individual viseme instances;
- extracting, based on the timestamping, transition light cones;
- extracting, based on the timestamping, audio clips;
- associating, based on the timestamping, the transition light cones with the audio clips;
- parsing the audio clips into sentences;
- tagging the sentences with parts of speech;
- training a modulation model on the tagged sentences;
- training a light cone model for viseme transitions, and
- training an audio model on phoneme transitions.
A method for implementing a machine-generated avatar, comprising:
- receiving, an input utterance;
- generating one or more phoneme sequences based on the input utterance;
- generating phoneme matrices;
- matching, based on phoneme and duration, corresponding viseme clips to the phoneme matrices;
- generating transition matrices for phoneme pairs using a trained audio linear regression model;
- generating transition matrices for viseme frames based on a trained light cone linear regression model;
- modulating phoneme and transition matrices to generate correct intonation using a trained modulation model;
- generating an audio track based on modulated matrices;
- and
- combining the audio track with visual frames.
Dependencies are inferred from the claim text.