Back to results
US 11,769,481 B2Granted
Unsupervised alignment for text to speech synthesis using neural networks
AssigneeNVIDIA Corporation
InventorsKevin Shih, Jose Rafael Valle Gomes da Costa, Rohan Badlani, Adrian Lancucki, Wei Ping, Bryan Catanzaro
Life of this patent
Estimating the term…
Abstract
Generation of synthetic speech from an input text sequence may be difficult when durations of individual phonemes forming the input text sequence are unknown. A predominantly parallel process may model speech rhythm as a separate generative distribution such that phoneme duration may be sampled at inference. Additional information such as pitch or energy may also be sampled to provide improved diversity for synthetic speech generation.
Claims18
A computer-implemented method, comprising:
- determining, from a plurality of audio segments, respective phoneme durations, phoneme pitches, and phoneme energies;
- determining a first alignment between a sequence of text and a total speech duration corresponding to probable locations for the respective phoneme durations;
- determining a second alignment for an audio segment of synthesized speech based, at least in part, on a first distribution corresponding to the phoneme durations and the first alignment;
- and
- generating, for the sequence of text, an audio segment comprising a synthesized recitation of the sequence of text based, at least in part, on the second alignment and at least one of a second distribution corresponding to the phoneme pitches, or a third distribution corresponding to the phoneme energies.
2 more elements
A method, comprising:
- determining, from a plurality of audio samples including human speech, alignments between text of the plurality of audio samples, a duration of the plurality of audio samples, and at least one of a pitch of the audio samples or an energy of the audio samples;
- generating an alignment distribution based, at least in part, on the alignments;
- determining a soft alignment between a first text sequence from the text and mel-frames of the alignment distribution, the soft alignment normalizing probability distributions for the alignments across the duration of the plurality of audio samples;
- and
- determining a hard alignment between the first text sequence and the mel-frames of the alignment distribution, the hard alignment concentrating the probability distributions from the soft alignment to a symbol for the alignments across the duration of the plurality of audio samples;
- determining one or more vectors, based on the hard alignment, corresponding to one or more speaker characteristics;
- receiving a second text sequence;
- and
- generating, based at least in part on the second text sequence and the one or more vectors, a synthetic audio clip corresponding to the second text sequence.
6 more elements
A processor, comprising:
- one or more processing units to:
- receive an audio clip of human speech represented as a mel-spectrogram;
- determine an alignment matrix for the audio clip normalized to a probability distribution;
- apply, to the alignment matrix, a prior distribution to exclude pairs of phonemes and mel-frames in the audio clip from the alignment matrix;
- determine, from the alignment matrix, an alignment for a first text sequence within the audio clip and a plurality of mel-frames representing a duration of the audio clip;
- determine one or more distributions for a pitch and an energy associated with the audio clip;
- receive a second text sequence;
- and
- generate a second audio clip of the second text sequence based, at least in part, on the alignment and the one or more distributions.
6 more elements
Dependencies are inferred from the claim text.