System and method for style extraction in speech synthesis using neural networks and stored augmentations to simulate degraded speech characteristics

A method for style extraction in speech synthesis, the method implemented by a speech processing system and comprising:

• extracting, from input audio data corresponding to input speech, one or more content elements and one or more non-content elements comprising using stored augmentations to facilitate separation of the input speech into the one or more content elements and the one or more non-content elements, wherein the one or more non-content elements comprise at least an input pitch and the stored augmentations simulate degraded speech characteristics and comprise one or more of background noise, air conditioner sounds, fan sounds, environmental sounds, masked data, microphone pops, smooth speech, or convolving speech;
• applying a neural network to encode the input pitch extracted from the input audio data without the one or more content elements in a low-dimensional vector, wherein the excluded one or more content elements comprise audio related to the content of what is being said by a speaker of the input speech including phonemes;
• combining the one or more content elements and the one or more non-content elements based on the low-dimensional vector to generate a new representation of the input speech;
• and
• providing output audio data generated based on the new representation of the input speech, wherein the output audio data comprises a pitch-consistent reconstruction of the input speech.

A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to:

• extract, from input audio data corresponding to input speech, one or more content elements and one or more non-content elements comprising using stored augmentations to facilitate separation of the input speech into the one or more content elements and the one or more non-content elements, wherein the one or more non-content elements comprise at least an input pitch and the stored augmentations simulate degraded speech characteristics and comprise one or more of background noise, air conditioner sounds, fan sounds, environmental sounds, masked data, microphone pops, smooth speech, or convolving speech;
• apply a neural network to encode the input pitch extracted from the input audio data in a latent representation, wherein the latent representation excludes the one or more content elements and the excluded one or more content elements comprise audio related to the content of what is being said by a speaker of the input speech including phonemes;
• combine the one or more content elements and the one or more non-content elements based on the latent representation to generate a new representation of the input speech;
• and
• provide output audio data generated based on the new representation of the input speech, wherein the output audio data preserves one or more pitch characteristics of the input speech.