# US 2021/0233550 A1 · Voice separation device, voice separation method, voice separation program, and voice separation system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233550 A1
- Document ID: US20210233550A1
- Status: Published application
- Kind: A1
- Application number: 17/146,834
- Filed: 2021-01-12
- Published: 2021-07-29
- Earliest claimed: 2018-08-24
- Assignees: Mitsubishi Electric Corporation; Mitsubishi Electric Research Laboratories, Inc.
- Applicants: Mitsubishi Electric Corporation; Mitsubishi Electric Research Laboratories, Inc.
- Inventors: Ryo AIHARA; Toshiyuki HANAZAWA; Yohei OKATO; Gordon P WICHERN; Jonathan LE ROUX
- Web page: https://patentgrep.com/patent/US20210233550A1

## CPC

- G10L 21/0272: Voice signal separating
- G10L 21/028: using properties of sound source
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/063: {Training}
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/30: using neural networks
- G10L 2021/02087: {the noise being separate speech, e.g. cocktail party}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A speech separation device (12) of a speech separation system includes a feature amount extraction unit (121) configured to extract time-series data of a speech feature amount of mixed speech, a block division unit (122) configured to divide the time-series data of the speech feature amount into blocks having a certain time width, a speech separation neural network (1b) configured to create time-series data of a mask of each of a plurality of speakers from the time-series data of the speech feature amount divided into blocks, and a speech restoration unit (123) configured to restore the speech data of each of the plurality of speakers from the time-series data of the mask and the time-series data of the speech feature amount of the mixed speech.

## Claims

### Claim 1

A speech separation device configured to separate, from mixed speech data of mixed speech including speech of a plurality of speakers, speech data of speech of each of the plurality of speakers, the speech separation device comprising:
feature amount extracting circuitry configured to extract time-series data of a speech feature amount of the mixed speech from the mixed speech data;
a block divider configured to divide the time-series data of the speech feature amount into blocks having a certain time width to generate time-series data of the speech feature amount divided into blocks;
a speech separation neural network including a combination of an LSTM neural network forward in a time axis direction and an LSTM neural network backward in the time axis direction, the speech separation neural network being configured to create time-series data of a mask of each of the plurality of speakers from the time-series data of the speech feature amount divided into blocks; and
speech restorating circuitry configured to restore the speech data of the speech of each of the plurality of speakers from the time-series data of the mask and the time-series data of the speech feature amount of the mixed speech,
the speech separation neural network being configured to use, when creating the time-series data of the mask of each of the plurality of speakers, the time-series data of a block earlier in time than a present time in the forward LSTM neural network, and the time-series data of a block composed of a predetermined number of frames later in time than the present time in the backward LSTM neural network.

### Claim 2 (depends on 1)

The speech separation device according to claim 1, further comprising a block length selector configured to select, when the time-series data of the speech feature amount is divided into one or more blocks in the time axis direction, a block length specifying the time width of the blocks in accordance with the speech feature amount,
wherein the speech separation neural network includes a plurality of speech separation neural networks, and
wherein the time-series data of the mask of each of the plurality of speakers is created by using, of the plurality of speech separation neural networks, a speech separation neural network corresponding to the block length.

### Claim 3 (depends on 1)

A speech separation system, comprising:
the speech separation device of claim 1; and
a speech separation model learning device configured to learn the speech separation neural network,
the speech separation model learning device including:
mixed speech creating circuitry configured to create, from learning data including speech data of individual speech uttered by each of a plurality of speakers, speech data of mixed speech including speech of the plurality of speakers;
feature amount extracting circuitry for learning configured to extract time-series data of a speech feature amount of the mixed speech from the speech data of the mixed speech and to extract time-series data of a speech feature amount of the individual speech from the speech data of the individual speech;
mask creating circuitry configured to create time-series data of a mask of each of the plurality of speakers from the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are extracted by the feature amount extracting circuitry for learning; and
model learning circuitry configured to learn the speech separation neural network from the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are extracted by the feature amount extracting circuitry for learning and from the time-series data of the mask created by the mask creating circuitry.

### Claim 4 (depends on 2)

A speech separation system, comprising:
the speech separation device of claim 2; and
a speech separation model learning device configured to learn the speech separation neural network,
the speech separation model learning device including:
mixed speech creating circuitry configured to create, from learning data including speech data of individual speech uttered by each of a plurality of speakers, speech data of mixed speech including speech of the plurality of speakers;
feature amount extracting circuitry for learning configured to extract time-series data of a speech feature amount of the mixed speech from the speech data of the mixed speech and to extract time-series data of a speech feature amount of the individual speech from the speech data of the individual speech;
mask creating circuitry configured to create time-series data of a mask of each of the plurality of speakers from the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are extracted by the feature amount extracting circuitry for learning; and
model learning circuitry configured to learn the speech separation neural network from the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are extracted by the feature amount extracting circuitry for learning and from the time-series data of the mask created by the mask creating circuitry.

### Claim 5 (depends on 3)

The speech separation system according to claim 3,
wherein the speech separation model learning device further includes a block divider for learning configured to divide the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are created by the mixed speech creating circuitry, and the time-series data of the mask created by the mask creating circuitry into one or more blocks in the time axis direction, and
wherein the model learning circuitry is configured to learn the speech separation neural network from the time-series data of the speech feature amount of the mixed speech divided into blocks by the block divider for learning, the time-series data of the speech feature amount of the individual speech divided into blocks by the block divider for learning, and the time-series data of the mask divided into blocks by the block divider for learning.

### Claim 6 (depends on 4)

The speech separation system according to claim 4,
wherein the speech separation model learning device further includes a block divider for learning configured to divide the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are created by the mixed speech creating circuitry, and the time-series data of the mask created by the mask creating circuitry into one or more blocks in the time axis direction, and
wherein the model learning circuitry is configured to learn the speech separation neural network from the time-series data of the speech feature amount of the mixed speech divided into blocks by the block divider for learning, the time-series data of the speech feature amount of the individual speech divided into blocks by the block divider for learning, and the time-series data of the mask divided into blocks by the block divider for learning.

### Claim 7 (depends on 1)

A speech separation system, comprising:
the speech separation device of claim 1; and
a speech separation model learning device configured to learn the speech separation neural network,
the speech separation model learning device including:
mixed speech creating circuitry configured to create, from learning data including speech data of individual speech uttered by each of a plurality of speakers, speech data of mixed speech including speech of the plurality of speakers;
feature amount extracting circuitry for learning configured to extract time-series data of a speech feature amount of the mixed speech from the speech data of the mixed speech and to extract time-series data of a speech feature amount of the individual speech from the speech data of the individual speech;
mask creating circuitry configured to create time-series data of a mask of each of the plurality of speakers from the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are extracted by the feature amount extracting circuitry for learning;
model learning circuitry configured to learn a teacher speech separation neural network from the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are extracted by the feature amount extracting circuitry for learning and from the time-series data of the mask created by the mask creating circuitry; and
student neural network learning circuitry configured to learn the speech separation neural network, which is a student speech separation neural network,
the student speech separation neural network being learned from the time-series data of the speech feature amount of the mixed speech, the time-series data of the speech feature amount of the individual speech, and the time-series data of the mask created by the teacher speech separation neural network.

### Claim 8 (depends on 2)

A speech separation system, comprising:
the speech separation device of claim 2; and
a speech separation model learning device configured to learn the speech separation neural network,
the speech separation model learning device including:
mixed speech creating circuitry configured to create, from learning data including speech data of individual speech uttered by each of a plurality of speakers, speech data of mixed speech including speech of the plurality of speakers;
feature amount extracting circuitry for learning configured to extract time-series data of a speech feature amount of the mixed speech from the speech data of the mixed speech and to extract time-series data of a speech feature amount of the individual speech from the speech data of the individual speech;
mask creating circuitry configured to create time-series data of a mask of each of the plurality of speakers from the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are extracted by the feature amount extracting circuitry for learning;
model learning circuitry configured to learn a teacher speech separation neural network from the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are extracted by the feature amount extracting circuitry for learning and from the time-series data of the mask created by the mask creating circuitry; and
student neural network learning circuitry configured to learn the speech separation neural network, which is a student speech separation neural network,
the student speech separation neural network being learned from the time-series data of the speech feature amount of the mixed speech, the time-series data of the speech feature amount of the individual speech, and the time-series data of the mask created by the teacher speech separation neural network.

### Claim 9 (depends on 7)

The speech separation system according to claim 7,
wherein the speech separation model learning device further includes a block divider for learning configured to divide the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are created by the mixed speech creating circuitry, and the time-series data of the mask created by the mask creating circuitry into one or more blocks in the time axis direction, and
wherein the student neural network learning circuitry is configured to learn the student speech separation neural network from the time-series data of the speech feature amount of the mixed speech divided into blocks by the block divider for learning, the time-series data of the speech feature amount of the individual speech divided into blocks by the block divider for learning, and the time-series data of the mask divided into blocks by the block divider for learning.

### Claim 10 (depends on 8)

The speech separation system according to claim 8,
wherein the speech separation model learning device further includes a block divider for learning configured to divide the time-series data of the speech feature amount of the mixed speech and the time-series data of the speech feature amount of the individual speech which are created by the mixed speech creating circuitry, and the time-series data of the mask created by the mask creating circuitry into one or more blocks in the time axis direction, and
wherein the student neural network learning circuitry is configured to learn the student speech separation neural network from the time-series data of the speech feature amount of the mixed speech divided into blocks by the block divider for learning, the time-series data of the speech feature amount of the individual speech divided into blocks by the block divider for learning, and the time-series data of the mask divided into blocks by the block divider for learning.

### Claim 11 (depends on 3)

The speech separation system according to claim 3, wherein the speech separation neural network includes:
an input layer to which the speech feature amount of the mixed speech is to be input;
an intermediate layer configured to create at least one of the mask or an embedded vector of the speech feature amount of the mixed speech;
a mask output layer configured to output the mask created by the intermediate layer; and
an embedded vector output layer configured to output the embedded vector.

### Claim 12 (depends on 4)

The speech separation system according to claim 4, wherein the speech separation neural network includes:
an input layer to which the speech feature amount of the mixed speech is to be input;
an intermediate layer configured to create at least one of the mask or an embedded vector of the speech feature amount of the mixed speech;
a mask output layer configured to output the mask created by the intermediate layer; and
an embedded vector output layer configured to output the embedded vector.

### Claim 13 (depends on 5)

The speech separation system according to claim 5, wherein the speech separation neural network includes:
an input layer to which the speech feature amount of the mixed speech is to be input;
an intermediate layer configured to create at least one of the mask or an embedded vector of the speech feature amount of the mixed speech;
a mask output layer configured to output the mask created by the intermediate layer; and
an embedded vector output layer configured to output the embedded vector.

### Claim 14 (depends on 6)

The speech separation system according to claim 6, wherein the speech separation neural network includes:
an input layer to which the speech feature amount of the mixed speech is to be input;
an intermediate layer configured to create at least one of the mask or an embedded vector of the speech feature amount of the mixed speech;
a mask output layer configured to output the mask created by the intermediate layer; and
an embedded vector output layer configured to output the embedded vector.

### Claim 15 (depends on 7)

The speech separation system according to claim 7, wherein the speech separation neural network includes:
an input layer to which the speech feature amount of the mixed speech is to be input;
an intermediate layer configured to create at least one of the mask or an embedded vector of the speech feature amount of the mixed speech;
a mask output layer configured to output the mask created by the intermediate layer; and
an embedded vector output layer configured to output the embedded vector.

### Claim 16 (depends on 8)

The speech separation system according to claim 8, wherein the speech separation neural network includes:
an input layer to which the speech feature amount of the mixed speech is to be input;
an intermediate layer configured to create at least one of the mask or an embedded vector of the speech feature amount of the mixed speech;
a mask output layer configured to output the mask created by the intermediate layer; and
an embedded vector output layer configured to output the embedded vector.

### Claim 17 (depends on 9)

The speech separation system according to claim 9, wherein the speech separation neural network includes:
an input layer to which the speech feature amount of the mixed speech is to be input;
an intermediate layer configured to create at least one of the mask or an embedded vector of the speech feature amount of the mixed speech;
a mask output layer configured to output the mask created by the intermediate layer; and
an embedded vector output layer configured to output the embedded vector.

### Claim 18 (depends on 10)

The speech separation system according to claim 10, wherein the speech separation neural network includes:
an input layer to which the speech feature amount of the mixed speech is to be input;
an intermediate layer configured to create at least one of the mask or an embedded vector of the speech feature amount of the mixed speech;
a mask output layer configured to output the mask created by the intermediate layer; and
an embedded vector output layer configured to output the embedded vector.

### Claim 19

A speech separation method for separating, from mixed speech data of mixed speech including speech of a plurality of speakers, speech data of speech of each of the plurality of speakers, the speech separation method comprising:
extracting time-series data of a speech feature amount of the mixed speech from the mixed speech data;
dividing the time-series data of the speech feature amount into blocks having a certain time width to generate time-series data of the speech feature amount divided into blocks;
creating, by a speech separation neural network including a combination of an LSTM neural network forward in a time axis direction and an LSTM neural network backward in the time axis direction, time-series data of a mask of each of the plurality of speakers from the time-series data of the speech feature amount divided into blocks; and
restoring the speech data of the speech of each of the plurality of speakers from the time-series data of the mask and the time-series data of the speech feature amount of the mixed speech,
the speech separation neural network being configured to use, when creating the time-series data of the mask of each of the plurality of speakers, the time-series data of a block earlier in time than a present time in the forward LSTM neural network, and the time-series data of a block composed of a predetermined number of frames later in time than the present time in the backward LSTM neural network.

### Claim 20

A speech separation program for causing a computer to execute processing of separating, from mixed speech data of mixed speech including speech of a plurality of speakers, speech data of speech of each of the plurality of speakers, the speech separation program causing the computer to execute:
feature amount extraction processing of extracting time-series data of a speech feature amount of the mixed speech from the mixed speech data;
block division processing of dividing the time-series data of the speech feature amount into blocks having a certain time width to generate time-series data of the speech feature amount divided into blocks;
mask creation processing of creating, by a speech separation neural network including a combination of an LSTM neural network forward in a time axis direction and an LSTM neural network backward in the time axis direction, time-series data of a mask of each of the plurality of speakers from the time-series data of the speech feature amount divided into blocks, the speech separation neural network being configured to use, when creating the time-series data of the mask of each of the plurality of speakers, the time-series data of a block earlier in time than a present time in the forward LSTM neural network, and the time-series data of a block composed of a predetermined number of frames later in time than the present time in the backward LSTM neural network; and
speech restoration processing of restoring the speech data of the speech of each of the plurality of speakers from the time-series data of the mask and the time-series data of the speech feature amount of the mixed speech.
