US 10,521,946 B1Granted

Processing speech to drive animations on avatars

AssigneeAmazon Technologies, Inc.

InventorsKyle Michael Roche, David Chiapperino, Timothy Kellogg, Christopher Russell Chiappone, Christine Morten

Life of this patent

Estimating the term…

Loading patent…

Abstract

Functionality is disclosed herein for using a framework for a VR/AR application to utilize different services. In some configurations, a VR/AR application can utilize different services, such as an animation service, a multi-modal disambiguation service, a virtual platform service, a recognition service, an automatic speech recognition (ASR) service, a text-to-speech (TTS) service, a search service, as well as one or more other services. Instead of a developer of the VR/AR application having to develop programming code to implement features provided by one or more of services, the developer may utilize functionality of existing services that are available from a service provider network.

Claims20

Dependent claims

A computer-implemented method comprising:

receiving text to be spoken during animation of an avatar;
determining, using a text-to-speech service, audio data associated with the text;
determining speech markup data including at least one or more phonic symbols, one or more words, and one or more time codes, the one or more time codes indicating a time of occurrence of the one or more phonic symbols and the one or more words during playback of the audio data;
generating one or more first animation sequences of a mouth of the avatar based at least in part on the one or more phonic symbols and the one or more time codes;
determining a location of an object in a virtual reality environment, the avatar depicted in the virtual reality environment;
generating one or more second animation sequences of at least one of a body, a head, or an appendage of the avatar based at least in part on the one or more words, the location of the object, and the one or more time codes;
and
outputting the audio data synchronized with output of combined animation sequences of the avatar that include the one or more first animation sequences of the mouth and the one or more second animation sequences of at least one of the body of the avatar or the appendage of the avatar.

5 more elements

A system comprising:

one or more computing devices operative to:
determine a location of an object in a virtual reality environment;
receive audio data that includes speech for an avatar depicted in the virtual reality environment;
determine speech markup data generated for the audio data, the speech markup data including at least:
one or more phonic symbols of sounds associated with one or more first time codes that indicate a time of occurrence of the sounds during playback of the audio data, and
one or more words associated with one or more second time codes that indicate a time of playback of the one or more words during playback of the audio data;
select one or more first animation sequences of one or more facial features of the avatar based at least in part on the one or more phonic symbols and the one or more first time codes;
select one or more second animation sequences of one or more skeletal components of the avatar including at least a torso and arms of the avatar based at least in part on the one or more words, the location of the object, and the one or more second time codes;
generate combined animation sequences of the avatar that include the one or more first animation sequences and the one or more second animation sequences, the combined animation sequences associated for playback at one or more third time codes based at least in part on the one or more first time codes and the one or more second time codes;
and
output at least a portion of the audio data synchronized with output of at least a portion of the combined animation sequences of the avatar.

8 more elements

Dependencies are inferred from the claim text.