# US 11,445,295 B2 · Low-latency speech separation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,295 B2 - Document ID: US11445295B2 - Status: Granted - Kind: B2 - Application number: 16/950,163 - Filed: 2020-11-17 - Published: 2021-03-11 - Granted: 2022-09-13 - Earliest claimed: 2019-04-05 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Zhuo Chen; Changliang Liu; Takuya Yoshioka; Xiong Xiao; Hakan Erdogan; Dimitrios Basile Dimitriadis - Web page: https://patentgrep.com/patent/US11445295B2 ## CPC - G10L 21/0272: Voice signal separating - H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)} - G10L 25/30: using neural networks - H04R 1/406: {microphones} - G10L 2021/02087: {the noise being separate speech, e.g. cocktail party} - G10L 2021/02166: {Microphone arrays; Beamforming} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract A system and method include reception of a first plurality of audio signals, generation of a second plurality of beamformed audio signals based on the first plurality of audio signals, each of the second plurality of beamformed audio signals associated with a respective one of a second plurality of beamformer directions, generation of a first TF mask for a first output channel based on the first plurality of audio signals, determination of a first beamformer direction associated with a first target sound source based on the first TF mask, generation of first features based on the first beamformer direction and the first plurality of audio signals, determination of a second TF mask based on the first features, and application of the second TF mask to one of the second plurality of beamformed audio signals associated with the first beamformer direction. ## Claims ### Claim 1 A computing system comprising: one or more processing units to execute processor-executable program code to cause the computing system to: receive a first plurality of audio signals; determine a first beamformer direction associated with a first target sound source based on the first plurality of audio signals; generate a second plurality of beamformed audio signals based on the first plurality of audio signals, each of the second plurality of beamformed audio signals associated with a respective one of a second plurality of beamformer directions; generate first features based on the first beamformer direction and the first plurality of audio signals; determine a Time Frequency (TF) mask based on the first features; and determine one of the second plurality of beamformed audio signals which is associated with the first beamformer direction; apply the TF mask to the one of the second plurality of beamformed audio signals associated with the first beamformer direction. ### Claim 2 (depends on 1) A computing system according to claim 1, the one or more processing units to execute processor-executable program code to cause the computing system to: determine a second beamformer direction associated with a second target sound source based on the based on the first plurality of audio signals; generate second features based on the second beamformer direction and the first plurality of audio signals; determine a second TF mask based on the second features; determine a second one of the second plurality of beamformed audio signals associated with the second beamformer direction; and apply the second TF mask to the second one of the second plurality of beamformed audio signals associated with the second beamformer direction. ### Claim 3 (depends on 2) A computing system according to claim 2, the one or more processing units to execute processor-executable program code to cause the computing system to: determine a third beamformer direction associated with a first interfering sound source based on the TF mask; generate the first features based on one of the second plurality of beamformed audio signals associated with the first beamformer direction, one of the second plurality of beamformed audio signals associated with the third beamformer direction, and the first plurality of audio signals; determine a fourth beamformer direction associated with a second interfering sound source based on the first plurality of audio signals; and generate the second features based on one of the second plurality of beamformed audio signals associated with the second beamformer direction, one of the second plurality of beamformed audio signals associated with the fourth beamformer direction, and the first plurality of audio signals. ### Claim 4 (depends on 3) A computing system according to claim 3, wherein the second plurality of beamformed audio signals are generated by a second plurality of fixed beamformers. ### Claim 5 (depends on 1) A computing system according to claim 1, wherein the second plurality of beamformed audio signals are generated by a second plurality of fixed beamformers. ### Claim 6 (depends on 1) A computing system according to claim 1, the one or more processing units to execute processor-executable program code to cause the computing system to: generate second features based on the first plurality of audio signals; and generate a second TF mask by inputting the second features to a trained neural network, wherein determination of the first beamformer direction associated with the first target sound source is based on the second TF mask and the first plurality of audio signals. ### Claim 7 (depends on 1) A computing system according to claim 1, wherein the TF mask associates each TF point of the first plurality of audio signals with a probability that the target sound source is a dominant sound source of the TF point. ### Claim 8 (depends on 1) A computing system according to claim 1, wherein application of the TF mask to the one of the second plurality of beamformed audio signals associated with the first beamformer direction generates an audio signal associated with the target sound source, the one or more processing units to execute processor-executable program code to cause the computing system to: perform speech recognition on the audio signal associated with the target sound source to generate a transcription. ### Claim 9 (depends on 2) A computing system according to claim 2, wherein application of the TF mask to the one of the second plurality of beamformed audio signals associated with the first beamformer direction generates an audio signal associated with the target sound source, and application of the second TF mask to the second one of the second plurality of beamformed audio signals associated with the second beamformer direction generates a second audio signal associated with the second target sound source, the one or more processing units to execute processor-executable program code to cause the computing system to: perform speech recognition on the audio signal associated with the target sound source and the second audio signal associated with the second target sound source to generate a transcription. ### Claim 10 A system comprising: a first plurality of fixed beamformers to receive a first plurality of audio signals and to generate a first plurality of beamformed audio signals based on the first plurality of audio signals, each of the first plurality of beamformed audio signals associated with a respective one of a first plurality of beamformer directions; a sound source localization component to determine a first beamformer direction associated with a first target sound source based on the first plurality of audio signals, and to determine one of the first plurality of beamformed audio signals which is associated with the first beamformer direction; a feature extraction component to generate first features based on one of the first plurality of beamformed audio signals associated with the first beamformer direction and the first plurality of audio signals; a Time Frequency (TF) mask generation network to generate a TF mask based on the first features; and a signal processing component to apply the TF mask to the one of the first plurality of beamformed audio signals associated with the first beamformer direction. ### Claim 11 (depends on 10) A system according to claim 10, the sound source localization component to determine a second beamformer direction associated with a second target sound source based on the based on the first plurality of audio signals and to determine a second one of the first plurality of beamformed audio signals associated with the second beamformer direction, the feature extraction component to generate second features based on the second beamformer direction and the first plurality of audio signals, the TF mask generation network determine a second TF mask based on the second features, and the signal processing component to apply the second TF mask to the second one of the first plurality of beamformed audio signals associated with the second beamformer direction. ### Claim 12 (depends on 11) A system according to claim 11, the sound source localization component to determine a third beamformer direction associated with a first interfering sound source based on the TF mask, and to determine a fourth beamformer direction associated with a second interfering sound source based on the first plurality of audio signals, the feature extraction component to generate the first features based on one of the first plurality of beamformed audio signals associated with the first beamformer direction, one of the first plurality of beamformed audio signals associated with the third beamformer direction, and the first plurality of audio signals, and the feature extraction component to generate the second features based on one of the first plurality of beamformed audio signals associated with the second beamformer direction, one of the first plurality of beamformed audio signals associated with the fourth beamformer direction, and the first plurality of audio signals. ### Claim 13 (depends on 10) A system according to claim 10, generate second features based on the first plurality of audio signals; and generate a second TF mask by inputting the second features to a trained neural network, wherein determination of the first beamformer direction associated with the first target sound source is based on the second TF mask and the first plurality of audio signals. ### Claim 14 (depends on 10) A system according to claim 10, wherein the TF mask associates each TF point of the first plurality of audio signals with a probability that the target sound source is a dominant sound source of the TF point. ### Claim 15 (depends on 10) A system according to claim 10, wherein application of the TF mask to the one of the first plurality of beamformed audio signals associated with the first beamformer direction generates an audio signal associated with the target sound source, the system further comprising: a speech recognition component to perform speech recognition on the audio signal associated with the target sound source to generate a transcription. ### Claim 16 (depends on 11) A system according to claim 11, wherein application of the TF mask to the one of the first plurality of beamformed audio signals associated with the first beamformer direction generates an audio signal associated with the target sound source, and application of the second TF mask to the second one of the first plurality of beamformed audio signals associated with the second beamformer direction generates a second audio signal associated with the second target sound source, the system comprising: a speech recognition component to perform speech recognition on the audio signal associated with the target sound source and the second audio signal associated with the second target sound source to generate a transcription. ### Claim 17 A computer-implemented method comprising: receiving a first plurality of audio signals; determining a first beamformer direction associated with a first target sound source based on the first plurality of audio signals; generating a second plurality of beamformed audio signals based on the first plurality of audio signals, each of the second plurality of beamformed audio signals associated with a respective one of a second plurality of beamformer directions; generating first features based on the first beamformer direction and the first plurality of audio signals; determining a Time Frequency (TF) mask based on the first features; and determining one of the second plurality of beamformed audio signals which is associated with the first beamformer direction; applying the TF mask to the one of the second plurality of beamformed audio signals associated with the first beamformer direction. ### Claim 18 (depends on 17) A computer-implemented method according to claim 17, further comprising: determining a second beamformer direction associated with a second target sound source based on the based on the first plurality of audio signals; generating second features based on the second beamformer direction and the first plurality of audio signals; determining a second TF mask based on the second features; determining a second one of the second plurality of beamformed audio signals associated with the second beamformer direction; and applying the second TF mask to the second one of the second plurality of beamformed audio signals associated with the second beamformer direction. ### Claim 19 (depends on 18) A computer-implemented method according to claim 18, further comprising: determining a third beamformer direction associated with a first interfering sound source based on the TF mask; generating the first features based on one of the second plurality of beamformed audio signals associated with the first beamformer direction, one of the second plurality of beamformed audio signals associated with the third beamformer direction, and the first plurality of audio signals; determining a fourth beamformer direction associated with a second interfering sound source based on the first plurality of audio signals; and generating the second features based on one of the second plurality of beamformed audio signals associated with the second beamformer direction, one of the second plurality of beamformed audio signals associated with the fourth beamformer direction, and the first plurality of audio signals.