# US 2020/0395041 A1 · Device, method, and program for analyzing speech signal

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0395041 A1
- Document ID: US20200395041A1
- Status: Published application
- Kind: A1
- Application number: 16/970,896
- Filed: 2019-02-19
- Published: 2020-12-17
- Earliest claimed: 2018-02-20
- Assignees: Nippon Telegraph and Telephone Corporation
- Applicants: Nippon Telegraph and Telephone Corporation
- Inventors: Ko TANAKA; Hirokazu KAMEOKA
- Web page: https://patentgrep.com/patent/US20200395041A1

## CPC

- G10L 25/69: for evaluating synthetic or decoded voice signals
- G10L 25/90: Pitch determination of speech signals
- G10L 15/142: {Hidden Markov Models [HMMs]}
- G10L 15/16: using artificial neural networks
- G10L 25/30: using neural networks
- G10L 25/75: for modelling vocal tract parameters
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A parameter included in a fundamental frequency pattern of a voice can be estimated from the fundamental frequency pattern with high accuracy and the fundamental frequency pattern of the voice can be reconstructed from the parameter included in the fundamental frequency pattern. A learning unit 30 learns a deep generation model including an encoder which regards a parameter included in a fundamental frequency pattern in a voice signal as a latent variable of the deep generation model and estimates the latent variable from the fundamental frequency pattern in the voice signal on the basis of parallel data of the fundamental frequency pattern in the voice signal and the parameter included in the fundamental frequency pattern in the voice signal, and a decoder which reconstructs the fundamental frequency pattern in the voice signal from the latent variable.

## Claims

### Claim at position 1

-7. (canceled)

### Claim 8

A computer-implemented method for estimating aspects of voice data, the method comprising:
learning a deep generation model, wherein the deep generation model comprises:
an encoder estimating a first parameter included in a first fundamental frequency pattern in a first voice data as a latent variable of the deep generation model using parallel data between the first fundamental frequency pattern in the first voice data and the first parameter included in the first fundamental frequency pattern in the first voice data, and
a decoder reconstructing the first fundamental frequency pattern in the first voice data based on the latent variable of the deep generation model;
estimating, based on a second fundamental frequency pattern in a second voice data, a second parameter included in the second fundamental frequency pattern using the encoder of the deep generation model; and
estimating, based on a third parameter included in a third fundamental frequency pattern in a third voice data, the third fundamental frequency pattern using the decoder of the deep generation model.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, the method further comprising:
maximizing an output of an objective function for learning the deep generation model, wherein the objective function is based at least on:
a distance between an output of the decoder having the first fundamental frequency pattern of the first voice data as an input and a prior distribution of the first parameter represented using a state sequence of a path-restricted hidden Markov model (HMM), and
an output of the encoder having the latent variable as an input of the decoder.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein each of the encoder and the decoder is configured using a convolutional neural network.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, wherein the first voice data is a learning data, wherein the first voice data and the second voice data are distinct, and wherein the first voice data and the third voice data are distinct.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9, wherein the first fundamental frequency pattern of the first voice data relates to one or more of:
an interrogative sentence based on the ending of an utterance sentence,
an intention of a speaker represented by the first voice data,
a melody of a singer represented by the first voice data, and
an emotion of the singer represented by the first voice data.

### Claim 13 (depends on 9)

The computer-implemented method of claim 9, wherein the first parameter in the first fundamental frequency pattern of the first voice data represents one or more of:
an accent of voice in the first voice data, and
vibrato and overshoot in the first voice data as a singing voice.

### Claim 14 (depends on 9)

The computer-implemented method of claim 9, the method further comprising:
receiving the first voice data;
receiving the first fundamental frequency pattern based on the received singing voice data;
generating the first parameter included in the first fundamental frequency pattern in the received singing voice data, wherein the first parameter relates to vibrato and overshoot;
learning the deep generation model; and
synthesizing voice data of the singer based on the learnt deep generation model; and
providing the synthesized voice data.

### Claim 15

A system for estimating aspects of voice data, the system comprises:
a processor; and
a memory storing computer-executable instructions that when executed by the processor cause the system to:
learn a deep generation model, wherein the deep generation model comprises:
an encoder estimating a first parameter included in a first fundamental frequency pattern in a first voice data as a latent variable of the deep generation model using parallel data between the first fundamental frequency pattern in the first voice data and the first parameter included in the first fundamental frequency pattern in the first voice data, and
a decoder reconstructing the first fundamental frequency pattern in the first voice data based on the latent variable of the deep generation model;
estimate, based on a second fundamental frequency pattern in a second voice data, a second parameter included in the second fundamental frequency pattern using the encoder of the deep generation model; and
estimate, based on a third parameter included in a third fundamental frequency pattern in a third voice data, the third fundamental frequency pattern using the decoder of the deep generation model.

### Claim 16 (depends on 15)

The system of claim 15, the computer-executable instructions when executed further causing the system to:
maximize an output of an objective function for learning the deep generation model, wherein the objective function is based at least on:
a distance between an output of the decoder having the first fundamental frequency pattern of the first voice data as an input and a prior distribution of the first parameter represented using a state sequence of a path-restricted hidden Markov model (HMM), and
an output of the encoder having the latent variable as an input of the decoder.

### Claim 17 (depends on 15)

The system of claim 15, wherein each of the encoder and the decoder is configured using a convolutional neural network.

### Claim 18 (depends on 15)

The system of claim 15, wherein the first voice data is a learning data, wherein the first voice data and the second voice data are distinct, and wherein the first voice data and the third voice data are distinct.

### Claim 19 (depends on 15)

The system of claim 15, wherein the first fundamental frequency pattern of the first voice data relates to one or more of:
an interrogative sentence based on the ending of an utterance sentence,
an intention of a speaker represented by the first voice data,
a melody of a singer represented by the first voice data, and
an emotion of the singer represented by the first voice data.

### Claim 20 (depends on 15)

The system of claim 15, wherein the first parameter in the first fundamental frequency pattern of the first voice data represents one or more of:
an accent of voice in the first voice data, and
vibrato and overshoot in the first voice data as a singing voice.

### Claim 21 (depends on 15)

The system of claim 15, the computer-executable instructions when executed further causing the system to:
receive the first voice data;
generate the first fundamental frequency pattern based on the received singing voice data;
generate the first parameter included in the first fundamental frequency pattern in the received singing voice data, wherein the first parameter relates to vibrato and overshoot;
learn the deep generation model; and
synthesize voice data of the singer based on the learnt deep generation model; and
provide the synthesized voice data.

### Claim 22

A computer-readable non-transitory recording medium storing computer-executable instructions that when executed by a processor cause a computer system to:
learn a deep generation model, wherein the deep generation model comprises:
an encoder estimating a first parameter included in a first fundamental frequency pattern in a first voice data as a latent variable of the deep generation model using parallel data between the first fundamental frequency pattern in the first voice data and the first parameter included in the first fundamental frequency pattern in the first voice data, and
a decoder reconstructing the first fundamental frequency pattern in the first voice data based on the latent variable of the deep generation model;
estimate, based on a second fundamental frequency pattern in a second voice data, a second parameter included in the second fundamental frequency pattern using the encoder of the deep generation model; and
estimate, based on a third parameter included in a third fundamental frequency pattern in a third voice data, the third fundamental frequency pattern using the decoder of the deep generation model.

### Claim 23 (depends on 22)

The computer-readable non-transitory recording medium of claim 22, the computer-executable instructions when executed further causing the system to:
maximize an output of an objective function for learning the deep generation model, wherein the objective function is based at least on:
a distance between an output of the decoder having the first fundamental frequency pattern of the first voice data as an input and a prior distribution of the first parameter represented using a state sequence of a path-restricted hidden Markov model (HMM), and
an output of the encoder having the latent variable as an input of the decoder.

### Claim 24 (depends on 22)

The computer-readable non-transitory recording medium of claim 22, wherein each of the encoder and the decoder is configured using a convolutional neural network.

### Claim 25 (depends on 22)

The computer-readable non-transitory recording medium of claim 22, wherein the first fundamental frequency pattern of the first voice data relates to one or more of:
an interrogative sentence based on the ending of an utterance sentence,
an intention of a speaker represented by the first voice data,
a melody of a singer represented by the first voice data, and
an emotion of the singer represented by the first voice data.

### Claim 26 (depends on 23)

The computer-readable non-transitory recording medium of claim 23, wherein the first parameter in the first fundamental frequency pattern of the first voice data represents one or more of:
an accent of voice in the first voice data, and
vibrato and overshoot in the first voice data as a singing voice.

### Claim 27 (depends on 23)

The computer-readable non-transitory recording medium of claim 23, the computer-executable instructions when executed further causing the system to:
receive the first voice data;
generate the first fundamental frequency pattern based on the received singing voice data;
generate the first parameter included in the first fundamental frequency pattern in the received singing voice data, wherein the first parameter relates to vibrato and overshoot;
learn the deep generation model; and
synthesize voice data of the singer based on the learnt deep generation model; and
provide the synthesized voice data.
