# US 2020/0395028 A1 · Audio conversion learning device, audio conversion device, method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0395028 A1
- Document ID: US20200395028A1
- Status: Published application
- Kind: A1
- Application number: 16/970,935
- Filed: 2019-02-20
- Published: 2020-12-17
- Earliest claimed: 2018-02-20
- Assignees: Nippon Telegraph and Telephone Corporation
- Applicants: Nippon Telegraph and Telephone Corporation
- Inventors: Hirokazu KAMEOKA; Takuhiro KANEKO; Ko TANAKA; Nobukatsu HOJO
- Web page: https://patentgrep.com/patent/US20200395028A1

## CPC

- G10L 21/003: Changing voice quality, e.g. pitch or formants
- G10L 21/007: characterised by the process used
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G10L 25/03: characterised by the type of extracted parameters
- G10L 2021/0135: {Voice conversion or morphing}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

To be able to convert to a voice of the desired attribution. Learning an encoder for, on the basis of parallel data of a sound feature vector series in a conversion-source voice signal and a latent vector series in the conversion-source voice signal, and an attribution label indicating attribution of the conversion-source voice signal, estimating a latent vector series from input of a sound feature vector series and an attribution label, and a decoder for reconfiguring the sound feature vector series from input of the latent vector series and the attribution label.

## Claims

### Claim at position 1

-8. (canceled)

### Claim 9

A computer-implemented method for converting aspects of voice, the method comprising:
receiving conversation source data;
generating a series of sound feature vectors based on the received conversation source data;
identifying a series of latent vectors based on the received conversation source data;
determining an attribution of the conversation source data;
generating an encoder based on training, wherein the encoder relates to determining a series of latent vectors and an attribution label of the conversation source voice data from the determined attribution based on parallel data between the generated series of latent vectors and the series of sound feature vectors; and
generating a decoder based on training, wherein the decoder relates to reconstructing the series of sound feature vectors of the conversation source data based on the received series of the latent vectors and the attribution label.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, the method further comprising:
generating the encoder and the decoder based on maximizing a value of an objective function,
wherein the objective function at least relates to an error between the reconstructed series of sound feature vectors and the generated series of sound feature vector as parallel data, and
wherein the objective function further relates to a distance between the determined series of latent vectors and the generated series of latent vectors as parallel data.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, wherein each of the encoder and the decoder is configured using one of: a convolutional network or a recurrent network.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9, wherein the attribution of the conversation source data includes one or more of:
gender of a speaker,
an status of the speaker being a native speaker of a language used,
a type of utterance mood of the speaker, and
a style of utterance in lecture or non-lecture.

### Claim 13 (depends on 9)

The computer-implemented method of claim 9, wherein the sound features vectors are based on one of:
a logarithmic amplitude spectrum;
a mel-cepstrum coefficient;
a linear predictive coefficient;
a Partial Correlation (PARCOR) coefficient; and
a Line Spectral Pair (LSP) parameter.

### Claim 14 (depends on 9)

The computer-implemented method of claim 9, the method further comprising:
receiving a voice data for a conversion of sound quality;
extracting an input series of sound feature vectors based on the voice data;
estimating, using the generated encoder, a series of latent vectors,
estimating, using the generated decoder, a series of target sound feature vectors for a target voice data based at least on the estimate series of latent vectors;
generating the target voice data based on the estimated series of target sound feature vectors; and
providing the generated target voice data as a converted voice data.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, wherein the generated target voice data relates to converting non-language aspects of the voice data while maintaining utterance sentences in the voice data, and wherein the non-language aspects of the voice include one or more of individuality and an utterance style of a speaker.

### Claim 16

A system for converting aspects of voice, the system comprises:
a processor; and
a memory storing computer-executable instructions that when executed by the processor cause the system to:
receive conversation source data;
generate a series of sound feature vectors based on the received conversation source data;
identify a series of latent vectors based on the received conversation source data;
determine an attribution of the conversation source data;
generate an encoder based on training, wherein the encoder relates to determining a series of latent vectors and an attribution label of the conversation source voice data from the determined attribution based on parallel data between the generated series of latent vectors and the series of sound feature vectors; and
generate an decoder based on training, wherein the decoder relates to reconstructing the series of sound feature vectors of the conversation source data based on the received series of the latent vectors and the attribution label.

### Claim 17 (depends on 16)

The system of claim 16, the computer-executable instructions when executed further causing the system to:
generate the encoder and the decoder based on maximizing a value of an objective function,
wherein the objective function at least relates to an error between the reconstructed series of sound feature vectors and the generated series of sound feature vector as parallel data, and
wherein the objective function further relates to a distance between the determined series of latent vectors and the generated series of latent vectors as parallel data.

### Claim 18 (depends on 16)

The system of claim 16, wherein each of the encoder and the decoder is configured using one of: a convolutional network or a recurrent network.

### Claim 19 (depends on 16)

The system of claim 16, wherein the attribution of the conversation source data includes one or more of:
gender of a speaker,
an status of the speaker being a native speaker of a language used,
a type of utterance mood of the speaker, and
a style of utterance in lecture or non-lecture.

### Claim 20 (depends on 16)

The system of claim 16, wherein the sound features vectors are based on one of:
a logarithmic amplitude spectrum;
a mel-cepstrum coefficient;
a linear predictive coefficient;
a Partial Correlation (PARCOR) coefficient; and
a Line Spectral Pair (LSP) parameter.

### Claim 21 (depends on 16)

The system of claim 16, the computer-executable instructions when executed further causing the system to:
receive a voice data for conversion;
extract an input series of sound feature vectors based on the voice data;
estimate, using the generated encoder, a series of latent vectors.
estimate, using the generated decoder, a series of target sound feature vectors for a target voice data based at least on the estimate series of latent vectors;
generate the target voice data based on the estimated series of target sound feature vectors; and
providing the generated target voice data as a converted voice data.

### Claim 22 (depends on 21)

The system of claim 21, wherein the generated target voice data relates to converting non-language aspects of the voice data while maintaining utterance sentences in the voice data, and wherein the non-language aspects of the voice include one or more of individuality and an utterance style of a speaker.

### Claim 23

A computer-readable non-transitory recording medium storing computer-executable instructions that when executed by a processor cause a computer system to:
receive conversation source data;
generate a series of sound feature vectors based on the received conversation source data;
identify a series of latent vectors based on the received conversation source data;
determine an attribution of the conversation source data;
generate an encoder based on training, wherein the encoder relates to determining a series of latent vectors and an attribution label of the conversation source voice data from the determined attribution based on parallel data between the generated series of latent vectors and the series of sound feature vectors; and
generate an decoder based on training, wherein the decoder relates to reconstructing the series of sound feature vectors of the conversation source data based on the received series of the latent vectors and the attribution label.

### Claim 24 (depends on 23)

The computer-readable non-transitory recording medium of claim 23, the computer-executable instructions when executed further causing the system to:
generate the encoder and the decoder based on maximizing a value of an objective function,
wherein the objective function at least relates to an error between the reconstructed series of sound feature vectors and the generated series of sound feature vector as parallel data, and
wherein the objective function further relates to a distance between the determined series of latent vectors and the generated series of latent vectors as parallel data.

### Claim 25 (depends on 23)

The computer-readable non-transitory recording medium of claim 23, wherein each of the encoder and the decoder is configured using one of: a convolutional network or a recurrent network.

### Claim 26 (depends on 23)

The computer-readable non-transitory recording medium of claim 23, wherein the attribution of the conversation source data includes one or more of:
gender of a speaker,
an status of the speaker being a native speaker of a language used,
a type of utterance mood of the speaker, and
a style of utterance in lecture or non-lecture.

### Claim 27 (depends on 23)

The computer-readable non-transitory recording medium of claim 23, the computer-executable instructions when executed further causing the system to:
receive a voice data for conversion;
extract an input series of sound feature vectors based on the voice data;
estimate, using the generated encoder, a series of latent vectors.
estimate, using the generated decoder, a series of target sound feature vectors for a target voice data based at least on the estimate series of latent vectors;
generate the target voice data based on the estimated series of target sound feature vectors; and
providing the generated target voice data as a converted voice data.

### Claim 28 (depends on 27)

The computer-readable non-transitory recording medium of claim 27,
wherein the generated target voice data relates to converting non-language aspects of the voice data while maintaining utterance sentences in the voice data, and wherein the non-language aspects of the voice include one or more of individuality and an utterance style of a speaker.
