Method and system for zero-shot speaker-adaptive speech synthesis
AssigneeNewSouth Innovations Pty Limited
InventorsSanjay Kumar Jha, Wenbin Wang, Yang Song
Life of this patent
Estimating the term…
Abstract
There is provided a method for synthesizing a speech waveform from text data. The method comprises: determining, from the text data, a phoneme sequence; obtaining a reference speech waveform comprising a high-level speech representation of a reference speaker speaking a reference speech; applying a trained neural network to the high-level speech representation to extract speaker embeddings; and determining, from the speaker embeddings and the phoneme sequence, a synthesized speech waveform indicative of the reference speaker speaking the text data.
Claims20
A method for synthesizing a speech waveform from text data, the method comprising:
- determining, from the text data, a phoneme sequence;
- determining, from the phoneme sequence, a timbre-invariant frame-level phoneme representation:
- obtaining a reference speech waveform comprising a high-level speech representation of a reference speaker speaking a reference speech;
- and
- applying a trained neural network to the high-level speech representation to:
- extract speaker embeddings;
- and
- determine, from the speaker embeddings and the timbre-invariant phoneme representation, a synthesized speech waveform indicative of the reference speaker speaking the text data.
A non-transitory computer readable medium comprising instructions stored thereon that, when executed by a processor, cause the processor to:
- determine, from a text data, a phoneme sequence;
- determine, from the phoneme sequence, a timbre-invariant frame-level phoneme representation:
- obtain a reference speech waveform comprising a high-level speech representation of a reference speaker speaking a reference speech;
- and
- apply a trained neural network to the high-level speech representation to:
- extract speaker embeddings;
- and
- determine, from the speaker embeddings and the timbre-invariant phoneme representation, a synthesized speech waveform indicative of the reference speaker speaking the text data.
A system for synthesizing a speech waveform from text data, the system comprising one or more processors configured to, individually or in combination;
- determine, from a text data, a phoneme sequence:
- determine, from the phoneme sequence, a timbre-invariant frame-level phoneme representation:
- obtain a reference speech waveform comprising a high-level speech representation of a reference speaker speaking a reference speech;
- and
- apply a trained neural network to the high-level speech representation to:
- extract speaker embeddings;
- and
- determine, from the speaker embeddings and the timbre-invariant phoneme representation, a synthesized speech waveform indicative of the reference speaker speaking the text data.
Dependencies are inferred from the claim text.