Systems and methods for speech separation and neural decoding of attentional selection in multi-speaker environments

A system comprising:

• at least one microphone to obtain a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;
• one or more neural sensors to obtain neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to;
• and
• a controller coupled to the at least one microphone and the one or more neural sensors, the controller configured to:
• generate a combined sound spectrogram from the combined sound signal;
• apply speech-separation processing to the combined sound spectrogram to derive multiple resultant speaker spectrograms that each corresponds to a different one of the multiple sound sources;
• and
• select one of the multiple resultant speaker spectrograms based on the obtained neural signals for the person, including to:
• generate an attended speaker spectrogram based on the neural signals for the person;
• compare the attended speaker spectrogram to the derived multiple resultant speaker spectrograms to select one of the multiple resultant speaker spectrograms;
• and
• transform the selected one of the multiple resultant speaker spectrograms into an acoustic signal;
• wherein the controller configured to apply the speech-separation processing to the combined sound spectrogram is configured to apply neural-network-based speech-separation processing to the combined sound spectrogram to derive the multiple resultant speaker spectrograms, including to:
• separate the combined sound spectrogram representation of the combined signal into the multiple resultant speaker spectrograms by feeding the sound spectrogram representation of the combined sound spectrogram into multiple deep neural networks (DNNs), each of which is trained to separate the combined sound spectrogram into a specific speaker spectrogram for a specific speaker from the multiple speakers, to obtain the multiple speaker spectrograms.

A non-transitory computer readable media programmed with instructions, executable on a processor, to:

• obtain, by a device comprising the processor, a combined sound signal for signals combined from multiple sound sources in an area in which a person is located;
• generate a combined sound spectrogram from the combined sound signal;
• apply, by the device, speech-separation processing to the combined sound spectrogram to derive multiple resultant speaker spectrograms that each corresponds to a different one of the multiple sound sources;
• obtain, by the device, neural signals for the person, the neural signals being indicative of one or more of the multiple sound sources the person is attentive to;
• and
• select one of the multiple resultant speaker spectrograms based on the obtained neural signals for the person, including to:
• generate an attended speaker spectrogram based on the neural signals for the person;
• compare the attended speaker spectrogram to the derived multiple resultant speaker spectrograms to select one of the multiple resultant speaker spectrograms;
• and
• transform the selected one of the multiple resultant speaker spectrograms into an acoustic signal;
• wherein the instructions to apply the speech-separation processing to the combined sound spectrogram comprise one or more instructions to apply neural-network-based speech-separation processing to the combined sound spectrogram to derive the multiple resultant speaker spectrograms, including to:
• separate the combined sound spectrogram representation of the combined signal into the multiple resultant speaker spectrograms by feeding the sound spectrogram representation of the combined sound spectrogram into multiple deep neural networks (DNNs), each of which is trained to separate the combined sound spectrogram into a specific speaker spectrogram for a specific speaker from the multiple speakers, to obtain the multiple speaker spectrograms.