US 2022/0028367 A1 · EXPRESSIVE TEXT-TO-SPEECH UTILIZING CONTEXTUAL WORD-LEVEL STYLE TOKENS

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate expressive audio for input texts based on a word-level analysis of the input text. For example, the disclosed systems can utilize a multi-channel neural network to generate a character-level feature vector and a word-level feature vector based on a plurality of characters of an input text and a plurality of words of the input text, respectively. In some embodiments, the disclosed systems utilize the neural network to generate the word-level feature vector based on contextual word-level style tokens that correspond to style features associated with the input text. Based on the character-level and word-level feature vectors, the disclosed systems can generate a context-based speech map. The disclosed systems can utilize the context-based speech map to generate expressive audio for the input text.

Claims20

Dependent claims

A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:

identify an input text comprising digital text having a plurality of characters and a plurality of words containing the plurality of characters;
generate a context-based speech map from the input text utilizing an expressive speech neural network having a multi-channel neural network architecture that encodes the plurality of characters and encodes the plurality of words containing the plurality of characters by:
determining, utilizing a character-level channel of the expressive speech neural network, a character-level feature vector based on a plurality of characters associated with the plurality of words;
determining, utilizing a word-level channel of the expressive speech neural network, a word-level feature vector based on contextual word embeddings corresponding to the plurality of words;
and
generating, utilizing a decoder of the expressive speech neural network, a context-based speech map based on the character-level feature vector and the word-level feature vector;
and
utilize the context-based speech map to generate expressive audio for the input text.

5 more elements

A system comprising:

one or more memory devices comprising:
an input text comprising digital text having a plurality of characters and a plurality of words containing the plurality of characters;
and
an expressive speech neural network having a multi-channel neural network architecture that includes a character-level channel, a word-level channel, and a decoder;
and
one or more server devices configured to cause the system to:
determine, utilizing the character-level channel of the expressive speech neural network, a character-level feature vector from character embeddings of the plurality of characters;
utilize the word-level channel of the expressive speech neural network to:
determine contextual word embeddings reflecting the plurality of words from the input text;
generate, utilizing an attention mechanism of the word-level channel, contextual word-level style tokens from the contextual word embeddings, the contextual word-level style tokens corresponding to different style features associated with the input text;
and
generate a word-level feature vector from the contextual word-level style tokens;
and
combine the character-level feature vector and the word-level feature vector utilizing the decoder to generate expressive audio for the input text.

11 more elements

Dependencies are inferred from the claim text.

Expressive text-to-speech utilizing contextual word-level style tokens

Life of this patent

Expressive text-to-speech utilizing contextual word-level style tokens

Life of this patent

Abstract

Claims20