Inferring emotion from speech in audio data using deep learning
AssigneeNVIDIA Corporation
InventorsIlia Federov, Dmitry Aleksandrovich Korobchenko
Life of this patent
Estimating the term…
Abstract
A deep neural network can be trained to infer emotion data from input audio. The network can be a transformer-based network that can infer probability values for a set of emotions or emotion classes. The emotion probability values can be modified using one or more heuristics, such as to provide for smoothing of emotion determinations over time, or via a user interface, where a user can modify emotion determinations as appropriate. A user may also provide prior emotion values to be blended with these emotion determination values. Determined emotion values can be provided as input to an emotion-based operation, such as to provide audio-driven speech animation.
Claims20
A computer-implemented method, comprising:
- computing, based at least on a neural network processing audio data representative of speech, a plurality of values indicative of emotions from a set of emotions in the speech;
- selecting, from the set of emotions, a subset of one or more emotions, the selecting being based at least on respective values from the plurality of values corresponding to the subset of one or more emotions exceeding a threshold;
- and
- rendering, based at least on a second neural network processing the audio data, data indicating the subset of one or more emotions, and one or more weighted values indicating one or more respective emotional strength values for the subset of the one or more emotions, an animation of a face such that one or more portions of the face move within the animation according to the speech and the subset of one or more emotions.
A processor comprising:
- one or more processing units to:
- provide audio data in an audio file format as input to a neural network;
- provide at least one style vector including instructions for rendering one or more facial components for one or more emotions;
- compute, based at least on the neural network processing the audio data, a plurality of values indicative of emotions from a set of emotions in the audio data;
- and
- render, based at least on a second neural network processing the audio data, data indicating a selected subset of one or more emotions, and one or more weighted values indicating one or more respective emotional strength values for the selected subset of the one or more emotions, an animation of the one or more facial components according to the audio data, the subset of one or more emotions, the weighted values indicating the one or more respective emotional strength values, and the at least one style vector.
A system comprising:
- one or more processing units to:
- compute, based at least on one or more neural networks processing audio data representative of speech, a plurality of values indicating a probability that emotions from a set of emotions correspond to the speech;
- compute, based at least on the one or more neural networks processing a selected subset of first values having respective values exceeding a threshold and having one or more weighted emotional strength values, a plurality of second values corresponding to a style of animation, and the audio data, a plurality of third values indicating one or more positions of one or more feature points corresponding to a virtual object;
- and
- render the virtual object based at least in part on the one or more third values.
Dependencies are inferred from the claim text.