US 12,014,748 B1Granted

Speech enhancement machine learning model for estimation of reverberation in a multi-task learning framework

AssigneeAmazon Technologies, Inc.

InventorsRitwik Giri, Mehmet Umut Isik, Neerad Dilip Phansalkar, Jean-Marc Valin, Karim Helwani, Arvindh Krishnaswamy

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques for training and using a machine learning model for estimation of reverberation in a multi-task learning framework are described. According to some embodiments, the multi-task learning framework improves the performance of the machine learning model by estimating the amount of reverberation present in an input audio recording as a secondary task to the primary task of generating a clean speech portion of the input audio recording. In one embodiment, a model architecture is selected that takes a noisy reverberant recording as an input and outputs an estimate of a clean (e.g., de-reverberated) signal, an estimate of noise (e.g., background noise), and an estimate of the reverb only portion, with the secondary task of estimating the reverb only portion acting as a regularizer that improves the machine learning model's performance in enhancing the reverberant (e.g., and noisy) input speech.

Claims19

Dependent claims

A computer-implemented method comprising:

receiving, at a machine learning service of a provider network, a plurality of training audio files and a request to create a machine learning model;
training, by the machine learning service of the provider network, an algorithm into the machine learning model that generates a clean speech portion of an audio file and a reverb only portion of the audio file;
generating, by the machine learning model, a reverb only portion and a clean speech portion of at least one of the plurality of training audio files;
determining a direct to reverberant ratio of the at least one of the plurality of training audio files based on the reverb only portion and the clean speech portion of the at least one of the plurality of training audio files;
filtering out the at least one of the plurality of training audio files having the direct to reverberant ratio below a reverberance threshold to generate a proper subset of the plurality of training audio files;
performing a training iteration with the proper subset of the plurality of training audio files to update the machine learning model;
receiving an inference request for an input audio file from a computing device of a user located outside the provider network;
generating, by the machine learning model, a clean speech portion of the input audio file and a reverb only portion of the input audio file;
generating an inference based at least in part on the clean speech portion of the input audio file and the reverb only portion of the input audio file;
and
transmitting the inference to a client application or to a storage location.

8 more elements

A computer-implemented method comprising:

receiving a plurality of training audio files and a request to create a machine learning model;
training an algorithm into the machine learning model that generates a reverb portion of an audio file;
generating, by the machine learning model, a reverb portion of at least one of the plurality of training audio files;
determining a direct to reverberant ratio of the at least one of the plurality of training audio files in part from the reverb portion of the at least one of the plurality of training audio files;
filtering out the at least one of the plurality of training audio files having the direct to reverberant ratio below a reverberance threshold to generate a proper subset of the plurality of training audio files;
performing a training iteration with the proper subset of the plurality of training audio files to generate an updated machine learning model;
receiving an inference request for an input audio file;
generating, by the machine learning model, a reverb portion of the input audio file;
generating an inference based at least in part on the reverb portion of the input audio file;
and
transmitting the inference to a client application or to a storage location.

8 more elements

A system comprising:

a first one or more electronic devices to implement a storage service in a provider network, the storage service to store a plurality of training audio files;
and
a second one or more electronic devices to implement a machine learning service in the provider network, the machine learning service including instructions that upon execution cause the machine learning service to perform operations comprising:
receiving a request to create a machine learning model,
training an algorithm into the machine learning model that generates a reverb portion of an audio file,
generating, by the machine learning model, a reverb portion of at least one of the plurality of training audio files,
determining a direct to reverberant ratio of the at least one of the plurality of training audio files in part from the reverb portion of the at least one of the plurality of training audio files,
filtering out the at least one of the plurality of training audio files having the direct to reverberant ratio below a reverberance threshold to generate a proper subset of the plurality of training audio files,
performing a training iteration with the proper subset of the plurality of training audio files to generate an updated machine learning model,
receiving an inference request for an input audio file,
generating, by the machine learning model, a reverb portion of the input audio file,
generating an inference based at least in part on the reverb portion of the input audio file, and
transmitting the inference to a client application or to a storage location.

10 more elements

Dependencies are inferred from the claim text.