Viseme data generation
AssigneeAmazon Technologies, Inc.
InventorsZoe Adams, Pete Klein, Derick Deller, Bradley Michael Richards, Anirudh Ranganath
Life of this patent
Estimating the term…
Abstract
Systems and methods for viseme data generation are disclosed. Uncompressed audio data is generated and/or utilized to determine the beats per minute of the audio data. Visemes are associated with the audio data utilizing a Viterbi algorithm and the beats per minute. A time-stamped list of viseme data is generated that associates the visemes with the portions of the audio data that they correspond to. An animatronic toy and/or an animation is caused to lip sync using the viseme data while audio corresponding to the audio data is output.
Claims20
A system comprising:
- one or more processors;
- and
- computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
- generating, in an uncompressed format, audio data corresponding to a song, the audio data representing frequencies and amplitudes of the song;
- analyzing the audio data to determine a number of beats per minute associated with the audio data;
- determining, using a Viterbi algorithm and the beats per minute, a portion of the audio data representing human sound;
- determining viseme data to associate with the audio data, the viseme data determined from an amplitude of the amplitudes;
- generating lip-sync data including the viseme data and the audio data;
- and
- sending the lip-sync data to a first device, the lip-sync data causing the first device to output the audio data and causing a second device to present the viseme data while the audio data is output.
A method, comprising:
- determining a portion of audio data representing human sound;
- determining amplitude data corresponding to one or more amplitudes of the portion of the audio data;
- determining viseme identifiers to associate with the audio data, the viseme identifiers determined based at least in part on the amplitude data;
- generating viseme data that associates the viseme identifiers and portions of the audio data corresponding to the viseme identifiers;
- generating at least one command associated with the viseme data, the at least one command configured to, when sent to a first device:
- cause, by a first processor of the first device, the first device to output the audio data, the first device being a voice-controlled device including a voice user interface;
- and
- cause, by a second processor of a second device corresponding to a different device than the first device, the second device to present a visual representation of the viseme identifiers while the audio data is output;
- and
- sending the viseme data, the audio data, and the at least one command to the first device.
A system, comprising:
- one or more processors;
- and
- computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
- receiving, at a first device and from a remote system, audio data;
- receiving, at the first device and from the remote system, viseme data indicating visemes associated with the audio data;
- receiving, at the first device and from the remote system, at least one command associated with the viseme data, the at least one command configured to cause a second device to present a visual representation of the viseme data while audio representing the audio data is output;
- determining that
- the second device is in communication with the first device;
- and
- causing the second device, via a first processor of the second device, to present a visual representation of the viseme data while a second processor of the first device causes the first device to output audio representing the audio data.
Dependencies are inferred from the claim text.