Text-Conditioned Speech Inpainting
AssigneeGoogle Inc.
InventorsZalán Borsos, Marco Tagliasacchi, Matthew Sharifi
Life of this patent
Estimating the term…
Abstract
Provided are systems, methods, and machine learning models for filling in gaps (e.g., of up to one second) in speech samples by leveraging an auxiliary textual input. Example machine learning models described herein can perform speech inpainting with the appropriate content, while maintaining speaker identity, prosody and recording environment conditions, and generalizing to unseen speakers. This approach significantly outperforms baselines constructed using adaptive TTS, as judged by human raters in side-by-side preference and MOS tests.
Claims20
A method for performing audio inpainting, the method comprising:
- receiving, by a computing system comprising one or more computing devices, an audio spectrogram, the audio spectrogram including at least one portion lacking audio content;
- receiving, by the computing system, a textual transcript associated with the audio spectrogram and including text corresponding to the at least one portion lacking audio content;
- processing, by the computing system, the audio spectrogram and the textual transcript with a machine-learned audio inpainting model to generate replacement audio content for the at least one portion of the audio spectrogram lacking audio content;
- and
- outputting, by the computing system, a completed audio spectrogram, the completed audio spectrogram having the replacement audio content in place of the at least one portion of the audio spectrogram lacking audio content.
A system for performing audio inpainting, the system comprising:
- one or more processors;
- and
- a non-transitory, computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
- receiving an audio spectrogram, the audio spectrogram including at least one portion lacking audio content;
- receiving a textual transcript associated with the audio spectrogram;
- processing the audio spectrogram and the textual transcript with a machine-learned audio inpainting model to generate replacement audio content for the at least one portion of the audio spectrogram lacking audio content;
- and
- outputting a completed audio spectrogram, the completed audio spectrogram having the replacement audio content in place of the at least one portion of the audio spectrogram lacking audio content.
A non-transitory, computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising:
- receiving a set of training data, the set of training data including a plurality of training audio spectrograms, each of the training audio spectrograms of the plurality of training audio spectrograms having an associated training textual transcript;
- generating a plurality of masked audio spectrograms by masking random consecutive frames of each training audio spectrogram of the plurality of training audio spectrograms, wherein each masked audio spectrogram of the plurality of masked audio spectrograms is associated with the training audio spectrogram used to generate the masked audio spectrogram and the training textual transcript associated with the training audio spectrogram;
- processing a masked audio spectrogram of the plurality of masked audio spectrograms and the training textual transcript associated with the masked audio spectrogram with a machine-learned audio inpainting model to generate an output spectrogram having replacement audio content in place of the masked frames of the masked audio spectrogram;
- evaluating a loss function that compares the output spectrogram with the training audio spectrogram associated with the masked audio spectrogram;
- and
- modifying one or more parameters of the machine-learned audio inpainting model based on the loss function.
Dependencies are inferred from the claim text.