System and method to use deep neural network to generate high-intelligibility binaural speech signals from single input

A system for generating binaural signals, the system comprising:

• a processing device, communicatively coupled to a microphone, to:
• receive a sound signal including speech and noise components;
• and
• transform, using a deep neural network (DNN), the sound signal into a first signal and a second signal, wherein the DNN comprises an encoding layer, a rendering layer, and a decoding layer, and to transform the sound signal, the processing device is further to:
• encode, using the encoding layer of the DNN, the sound signal into a sound signal representation in a latent space, wherein the latent space includes vectors representing features extracted from the sound signal;
• render, using the rendering layer of the DNN, the sound signal representation into a first signal representation and a second signal representation, wherein the rendering layer of the DNN comprises binaural rendering functions, and the DNN is trained to learn parameters of the binaural rendering functions based on a signal distortion index, wherein the processing device is further to:
• specify the signal distortion index for sound signals;
• receive a training dataset comprising a combined sequence of noisy signal data points, a first sequence of left-channel noisy signal data points, and a second sequence of right-channel noisy signal data points;
• calculate respective signal distortion index values for each of the combined sequence of noisy signal data points, the first sequence of left-channel noisy signal data points, and the second sequence of right-channel noisy signal data points;
• and
• update the parameters of the binaural rendering functions based on the signal distortion index values for each of the combined sequence of noisy signal data points, the first sequence of left-channel noisy signal data points, and the second sequence of right-channel noisy signal data points, wherein the processing device is further to:
• measure a duration of each of clean speech signals and each of noise signals;
• select, for each clean speech signal and a corresponding noise signal, a portion of the corresponding noise signal with a duration that is equal to a difference between the duration of the clean speech signal and the duration of the corresponding noise signal;
• trim the corresponding noise signal, wherein to trim the corresponding noise signal, the processing device is further to remove the selected portion of the corresponding noise signal based on the duration of the clean speech signal being shorter than the duration of the corresponding noise signal;
• and append a copy of the selected portion to the corresponding noise signal based on the duration of clean speech signal being shorter than the duration of the corresponding noise signal;
• and
• generate the combined sequence of noisy signal data points based on the clean speech signals and the trimmed corresponding noise signals;
• and
• decode, using the decoder layer of the DNN, the first signal representation into the first signal and the second signal representation into the second signal.

A non-transitory machine-readable storage medium storing instructions for generating binaural signals which, when executed, cause a processing device to:

• receive a sound signal including speech and noise components;
• and
• transform, using a deep neural network (DNN), the sound signal into a first signal and a second signal, wherein the DNN comprises an encoding layer, a rendering layer, and a decoding layer, and to transform the sound signal, the instructions, when executed, further cause the processing device to:
• encode, using the encoding layer of the DNN, the sound signal into a sound signal representation in a latent space, wherein the latent space includes vectors representing features extracted from the sound signal;
• render, using the rendering layer of the DNN, the sound signal representation into a first signal representation and a second signal representation in the latent space, wherein the rendering layer of the DNN comprises binaural rendering functions, and the DNN is trained to learn parameters of the binaural rendering functions based on a signal distortion index, wherein the processing device is further to:
• specify the signal distortion index for sound signals;
• receive a training dataset comprising a combined sequence of noisy signal data points, a first sequence of left-channel noisy signal data points, and a second sequence of right-channel noisy signal data points;
• calculate respective signal distortion index values for each of the combined sequence of noisy signal data points, the first sequence of left-channel noisy signal data points, and the second sequence of right-channel noisy signal data points;
• and
• update the parameters of the binaural rendering functions based on the signal distortion index values for each of the combined sequence of noisy signal data points, the first sequence of left-channel noisy signal data points, and the second sequence of right-channel noisy signal data points, wherein the processing device is further to:
• measure a duration of each of clean speech signals and each of noise signals;
• select, for each clean speech signal and a corresponding noise signal, a portion of the corresponding noise signal with a duration that is equal to a difference between the duration of the clean speech signal and the duration of the corresponding noise signal;
• trim the corresponding noise signal, wherein to trim the corresponding noise signal, the processing device is further to remove the selected portion of the corresponding noise signal based on the duration of the clean speech signal being shorter than the duration of the corresponding noise signal;
• and append a copy of the selected portion to the corresponding noise signal based on the duration of clean speech signal being shorter than the duration of the corresponding noise signal;
• and
• generate the combined sequence of noisy signal data points based on the clean speech signals and the trimmed corresponding noise signals;
• and
• decode, using the decoding layer of the DNN, the first signal representation into the first signal and the second signal representation into the second signal.