US 10,586,369 B1Granted

Using dialog and contextual data of a virtual reality environment to create metadata to drive avatar animation

AssigneeAmazon Technologies, Inc.

InventorsKyle Michael Roche, David Chiapperino, Christine Morten, Kathleen Alison Curry, Leo Chan

Life of this patent

Estimating the term…

Loading patent…

Abstract

One or more services may generate audio data and animations of an avatar based on input text. A speech input ingestion (SII) service may identify tags of objects in a virtual environment and associate tags of those objects with words in the input text, which may be stored as metadata in speech markup data. This association may enable an animation service to generate gestures toward objects while animating an avatar, or may be used to create animations or effects of the object. The SII service may analyze input text to identify dialog including multiple speakers associated with the text. The SII service may create metadata to associate certain words with respective speakers (avatars) of those words, which may be processed by the animation service to animate multiple avatars speaking the dialog.

Claims20

Dependent claims

A system comprising:

a recognition service to:
determine a first tag associated with a first object and a second tag associated with a second object, the first object and the second object located in a virtual environment;
a speech input ingestion service to:
analyze text in speech markup data to determine a word in the text that is associated with at least the first tag or the second tag;
determine that the word is associated with the first tag of the first object;
determine an object identifier of the first object;
create metadata for the object identifier;
and
associate the object identifier in the metadata with the word in the speech markup data;
a text-to-speech service to:
process the text to create audio data and phonics for words in the text including the word;
and
an animation service to:
process the phonics to create animations of an avatar, including at least animation of facial features synchronized with playback of the audio data;
and
process the metadata to cause a first animation of the avatar in the virtual environment in relation to the first object, wherein the process performs acts to:
determine a location of the first object in the virtual environment;
and
generate the first animation for the avatar to depict the avatar gesturing toward the location associated with the first object at a time associated with output of the word in the audio data.

16 more elements

A non-transitory computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by one or more computing devices, cause the one or more computing devices to:

determine a tag associated with an object located in a virtual environment;
analyze text, in speech markup data, to be spoken by an avatar during animation of the avatar in the virtual environment;
determine that a word in the text is associated with the tag of the object;
determine an identifier of the object, the identifier being associated with metadata;
associate the identifier with the word;
determine audio data and one or more phonics for one or more words in the text including the word;
determine, based at least in part on the one or more phonics, one or more animations of the avatar, the one or more animations including animation of one or more facial features synchronized with playback of the audio data;
and
cause, based at least in part on the metadata, an animation of the avatar in the virtual environment in relation to the object, wherein causing the animation of the avatar comprises:
determining a location of the object in the virtual environment;
and
generating the animation for the avatar to depict the avatar gesturing toward the location associated with the object at a time associated with output of the word in the audio data.

9 more elements

Dependencies are inferred from the claim text.