Back to results
US 2023/0419947 A1Published application
Unsupervised alignment for text to speech synthesis using neural networks
AssigneeNVIDIA Corporation
InventorsKevin Shih, Jose Rafael Valle Gomes da Costa, Rohan Badlani, Adrian Lancucki, Wei Ping, Bryan Catanzaro
Life of this patent
Estimating the term…
Abstract
Generation of synthetic speech from an input text sequence may be difficult when durations of individual phonemes forming the input text sequence are unknown. A predominantly parallel process may model speech rhythm as a separate generative distribution such that phoneme duration may be sampled at inference. Additional information such as pitch or energy may also be sampled to provide improved diversity for synthetic speech generation.
Claims20
A computer-implemented method, comprising:
- determining a phoneme distribution from respective phoneme durations of a plurality of first audio segments;
- determining, from at least respective phoneme pitches or respective phoneme energies of the plurality of first audio segments, at least a pitch distribution or an energy distribution;
- determining a duration alignment between a sequence of text and a total speech duration using the respective phoneme durations;
- and
- determining a speech alignment for a second audio segment based at least on the duration alignment, the phoneme distribution, and at least one of the pitch distribution or the energy distribution.
2 more elements
A system, comprising:
- one or more processing units to generate, for a sequence of text, a synthetic audio segment comprising a synthesized recitation of the sequence of text based, at least, on a synthetic alignment and at least one distribution, wherein the synthetic alignment is based on phoneme distributions for a plurality of audio segments and a first alignment between the sequence of text and a total speech duration, and further wherein the at least one distribution corresponds to respective phoneme pitches from the plurality of audio segments or respective phoneme energies from the plurality of audio segments.
A processor, comprising:
- one or more processing units to:
- determine, from a plurality of audio samples including human speech, alignments between text of the plurality of audio samples, a duration of the plurality of audio samples, and at least one of a pitch of the audio samples or an energy of the audio samples;
- generate an alignment distribution based on the alignments;
- determine one or more vectors corresponding to one or more speaker characteristics based on a concentrated probability distribution of a text sequence from the text and mel-frames of the alignment distribution across the duration of the plurality of audio samples.
1 more element
Dependencies are inferred from the claim text.