# US 2020/0349954 A1 · Processing Overlapping Speech from Distributed Devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0349954 A1
- Document ID: US20200349954A1
- Status: Published application
- Kind: A1
- Application number: 16/399,175
- Filed: 2019-04-30
- Published: 2020-11-05
- Earliest claimed: 2019-04-30
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Takuya Yoshioka; Andreas Stolcke; Zhuo Chen; Dimitrios Basile Dimitriadis; Nanshan Zeng; Lijuan Qin; William Isaac Hinthorn; Xuedong Huang
- Web page: https://patentgrep.com/patent/US20200349954A1

## CPC

- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 15/16: using artificial neural networks
- G10L 21/0208: Noise filtering
- G10L 21/0272: Voice signal separating
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2021/02087: {the noise being separate speech, e.g. cocktail party}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A computer implemented method includes receiving audio signals representative of speech via multiple audio streams transmitted from corresponding multiple distributed devices, performing, via a neural network model, continuous speech separation for one or more of the received audio signals having overlapped speech, and providing the separated speech on a fixed number of separate output audio channels.

## Claims

### Claim 1

A computer implemented method comprising:
receiving audio signals representative of speech via multiple audio streams transmitted from corresponding multiple distributed devices;
performing, via a neural network model, continuous speech separation for one or more of the received audio signals having overlapped speech; and
providing the separated speech on a fixed number of separate output audio channels.

### Claim 2 (depends on 1)

The method of claim 1 wherein performing continuous speech separation is performed by the neural network model trained using permutation invariant training.

### Claim 3 (depends on 2)

The method of claim 2 wherein the neural network model is configured to receive a varying number of inputs to support a dynamic change in a number of audio signals and locations of distributed devices during a meeting between multiple users.

### Claim 4 (depends on 1)

The method of claim 1 wherein the multiple devices capture the audio signals during an ad-hoc meeting.

### Claim 5 (depends on 1)

The method of claim 1 wherein the audio signals are received at a meeting server coupled to the distributed devices via a network.

### Claim 6 (depends on 1)

The method of claim 1 and further comprising generating a transcript based on the separate audio channels.

### Claim 7 (depends on 6)

The method of claim 6 and further comprising including speaker attribution in the generated transcript.

### Claim 8 (depends on 7)

The method of claim 7 and further comprising sending the transcript to one or more of the distributed devices.

### Claim 9 (depends on 1)

The method of claim 1 wherein at least two of the audio streams are provided by an ambient capture device having an array of microphones in fixed positions.

### Claim 10

A machine-readable storage device having instructions for execution by a processor of a machine to cause the processor to perform operations to perform a method, the operations comprising:
receiving audio signals representative of speech via multiple audio streams transmitted from corresponding multiple distributed devices;
performing, via a neural network model, continuous speech separation for one or more of the received audio signals having overlapped speech; and
providing the separated speech on a fixed number of separate output audio channels.

### Claim 11 (depends on 10)

The device of claim 10 wherein performing continuous speech separation is performed by a neural network model trained using permutation invariant training.

### Claim 12 (depends on 11)

The device of claim 11 wherein the neural network model is configured to receive a varying number of inputs to support a dynamic change in a number of audio signals and locations of distributed devices during a meeting between multiple users.

### Claim 13 (depends on 10)

The device of claim 10 wherein the multiple distributed devices comprise wireless devices associated with speakers in a meeting.

### Claim 14 (depends on 10)

The device of claim 10 wherein the audio signals are received at a meeting server coupled to the distributed devices via a network.

### Claim 15 (depends on 10)

The device of claim 10 and further comprising generating a speaker attributed transcript based on the separate audio channels.

### Claim 16 (depends on 15)

The device of claim 15 and further comprising sending the transcript to one or more of the distributed devices.

### Claim 17

The device of claim wherein at least two of the audio streams are provided by an ambient capture device having an array of microphones in fixed positions.

### Claim 18

A device comprising:
a processor; and
a memory device coupled to the processor and having a program stored thereon for execution by the processor to perform operations comprising:
receiving audio signals representative of speech via multiple audio streams transmitted from corresponding multiple distributed devices;
performing, via a neural network model, continuous speech separation for one or more of the received audio signals having overlapped speech; and
providing the separated speech on a fixed number of separate output audio channels.

### Claim 19 (depends on 18)

The device of claim 18 wherein performing continuous speech separation is performed by a neural network model trained using permutation invariant training and wherein the neural network model is configured to receive a varying number of inputs to support a dynamic change in a number of audio signals and locations of distributed devices during a meeting between multiple users.

### Claim 20 (depends on 18)

The device of claim 18 wherein the audio signals are received at a meeting server coupled to the distributed devices via a network, and wherein the meeting server performs addition operations comprising:
generating a speaker attributed transcript based on the separate audio channels; and
sending the transcript to one or more of the distributed devices.
