# US 2019/0318757 A1 · Multi-microphone speech separation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318757 A1
- Document ID: US20190318757A1
- Status: Published application
- Kind: A1
- Application number: 15/991,988
- Filed: 2018-05-29
- Published: 2019-10-17
- Earliest claimed: 2018-04-11
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Zhuo CHEN; Hakan ERDOGAN; Takuya YOSHIOKA; Fileno A. ALLEVA; Xiong XIAO
- Web page: https://patentgrep.com/patent/US20190318757A1

## CPC

- G10L 21/0272: Voice signal separating
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 15/16: using artificial neural networks
- G10L 17/04: Training, enrolment or model building
- G10L 17/18: Artificial neural networks; Connectionist approaches
- G10L 19/022: Blocking, i.e. grouping of samples in time; Choice of analysis windows; Overlap factoring
- G10L 21/0208: Noise filtering
- H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)}
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 17/00: Speaker identification or verification techniques
- G10L 2021/02087: {the noise being separate speech, e.g. cocktail party}
- G10L 2021/02166: {Microphone arrays; Beamforming}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

This document relates to separation of audio signals into speaker-specific signals. One example obtains features reflecting mixed speech signals captured by multiple microphones. The features can be input a neural network and masks can be obtained from the neural network. The masks can be applied one or more of the mixed speech signals captured by one or more of the microphones to obtain two or more separate speaker-specific speech signals, which can then be output.

## Claims

### Claim 1

A method performed on a computing device, the method comprising:
obtaining features reflecting mixed speech signals captured by multiple microphones;
inputting the features to a neural network;
obtaining masks output by the neural network;
applying the masks to at least one of the mixed speech signals captured by at least one of the microphones to obtain two or more separate speaker-specific speech signals; and
outputting the two or more separate speaker-specific speech signals.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
applying the masks by multiplying the masks by a power spectrogram recorded by an individual microphone.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
applying the masks by deriving beamformers from the masks and using the beamformers to obtain the two or more separate speaker-specific speech signals.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
gain adjusting the two or more separate speaker-specific speech signals prior to the outputting.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
training the neural network using permutation invariant training.

### Claim 6 (depends on 1)

The method of claim 1, the features comprising power spectra and phase features.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
normalizing the features before inputting the features to the neural network.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
inputting the features to the neural network as sliding windows, individual sliding windows comprising multiple audio segments;
obtaining the masks from the neural network, the masks being output for respective sliding windows; and
stitching masks for at least two adjacent sliding windows together.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
processing the respective mixed speech signals to obtain the features, the features comprising plurality of frequency bins for individual segments of audio from each microphone, each frequency bin comprising magnitude and phase values of a Fourier transform.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
combining the individual segments into the sliding windows, the sliding windows having respective overlapping portions.

### Claim 11

A system comprising:
a hardware processing unit; and
a storage resource storing computer-readable instructions which, when executed by the hardware processing unit, cause the hardware processing unit to:
obtain features reflecting multiple mixed speech signals captured by multiple microphones;
normalize the features to obtain normalized features;
input the normalized features to a speech separation model;
obtain, from the speech separation model, respective masks for individual speakers; and
apply the respective masks to at least one of the mixed speech signals to obtain at least two separate speaker-specific speech signals for different speakers.

### Claim 12 (depends on 11)

The system of claim 11, further comprising the multiple microphones.

### Claim 13 (depends on 12)

The system of claim 12, the multiple microphones being synchronized via a shared clock, wherein the shared clock is a local hardware clock or a logical clock synchronized over a network.

### Claim 14 (depends on 11)

The system of claim 11, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to:
perform automated speech recognition on the at least two separate speaker-specific signals to identify words spoken by the different speakers.

### Claim 15 (depends on 14)

The system of claim 14, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to:
produce a transcript identifying first words spoken by a first speaker and second words spoken by a second speaker.

### Claim 16 (depends on 14)

The system of claim 14, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to:
perform a first action in response to a first word spoken by a first speaker; and
perform a second action in response to a second word spoken by a second speaker.

### Claim 17 (depends on 11)

The system of claim 11, the features comprising power spectra and inter-microphone phase differences.

### Claim 18 (depends on 17)

The system of claim 17, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to:
perform mean and variance normalization on the power spectra; and
perform mean normalization without variance normalization on the inter-microphone phase differences.

### Claim 19

A computer-readable storage medium storing instructions which, when executed by a processing device, cause the processing device to perform acts comprising:
obtaining speaker-specific masks from a speech separation model, the speech separation model producing the speaker-specific masks from input of at least two different microphones;
using the speaker-specific masks to derive respective speaker-specific beamformed signals;
performing gain adjustment on the respective speaker-specific beamformed signals to obtain gain-adjusted speaker-specific beamformed signals; and
outputting the gain-adjusted speaker-specific beamformed signals.

### Claim 20 (depends on 19)

The computer-readable storage medium of claim 19, wherein the gain adjustment comprises modifying a gain of an individual speaker-specific beamformed signal of an individual speaker with a gain of a corresponding masked signal for the individual speaker.