# US 2021/0280202 A1 · Voice conversion method, electronic device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0280202 A1
- Document ID: US20210280202A1
- Status: Published application
- Kind: A1
- Application number: 17/330,126
- Filed: 2021-05-25
- Published: 2021-09-09
- Earliest claimed: 2020-09-25
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science and Technology Co., Ltd.
- Inventors: Xilei WANG; Wersfu WANG; Tao SUN
- Web page: https://patentgrep.com/patent/US20210280202A1

## CPC

- G10L 21/007: characterised by the process used
- G10L 21/013: Adapting to target pitch
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 21/003: Changing voice quality, e.g. pitch or formants
- G10L 25/30: using neural networks
- G10L 15/14: using statistical models, e.g. Hidden Markov Models [HMMs] (G10L15/18 takes precedence)
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L 2021/0135: {Voice conversion or morphing}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The disclosure provides a voice conversion method, a voice conversion apparatus, an electronic device, and a storage medium, related to the field of voice conversion, speech interaction, natural language processing, and deep learning. The method includes: acquiring a source speech of a first user and a reference speech of a second user; extracting first speech content information and a first acoustic feature from the source speech; extracting a second acoustic feature from the reference speech; acquiring a reconstructed third acoustic feature by inputting the first speech content information, the first acoustic feature, and the second acoustic feature into a pre-trained voice conversion model, in which the pre-trained voice conversion model is acquired by training based on speeches of a third user; and synthesizing a target speech based on the third acoustic feature.

## Claims

### Claim 1

A voice conversion method, comprising:
acquiring a source speech of a first user and a reference speech of a second user;
extracting first speech content information and a first acoustic feature from the source speech;
extracting a second acoustic feature from the reference speech;
acquiring a reconstructed third acoustic feature by inputting the first speech content information, the first acoustic feature, and the second acoustic feature into a pre-trained voice conversion model, in which the pre-trained voice conversion model is acquired by training based on speeches of a third user; and
synthesizing a target speech based on the third acoustic feature.

### Claim 2 (depends on 1)

The method as claimed in claim 1, wherein extracting the first speech content information from the source speech comprises:
acquiring a phonetic posterior gram by inputting the source speech into a pre-trained multilingual automatic speech recognition model; and
using the phonetic posterior gram as the first speech content information.

### Claim 3 (depends on 1)

The method as claimed in claim 1, wherein the first acoustic feature, the second acoustic feature, and the third acoustic feature are Mel features.

### Claim 4 (depends on 1)

The method as claimed in claim 1, wherein the voice conversion model comprises a hidden-variable network, a timbre network, and a reconstruction network, and acquiring the reconstructed third acoustic feature by inputting the first speech content information, the first acoustic feature, and the second acoustic feature into the pre-trained voice conversion model comprises:
acquiring a fundamental frequency and an energy parameter by inputting the first acoustic feature into the hidden-variable network;
acquiring a timbre parameter by inputting the second acoustic feature into the timbre network; and
acquiring the third acoustic feature by inputting the first speech content information, the fundamental frequency and energy parameter, and the timbre parameter into the reconstruction network.

### Claim 5 (depends on 4)

The method as claimed in claim 4, wherein acquiring the fundamental frequency and energy parameter by inputting the first acoustic feature into the hidden-variable network comprises:
inputting the first acoustic feature into the hidden-variable network, such that the hidden-variable network compresses the first acoustic feature on a frame scale, and extracts the fundamental frequency and energy parameter from the compressed first acoustic feature.

### Claim 6 (depends on 4)

The method as claimed in claim 4, wherein acquiring the timbre parameter by inputting the second acoustic feature into the timbre network comprises:
inputting the second acoustic feature into the timbre network, such that the timbre network abstracts the second acoustic feature by a deep recurrent neural network and a variational auto encoder to acquire the timbre parameter.

### Claim 7 (depends on 4)

The method as claimed in claim 4, wherein acquiring the third acoustic feature by inputting the first speech content information, the fundamental frequency and energy parameter, and the timbre parameter into the reconstruction network comprises:
inputting the first speech content information, the fundamental frequency and energy parameter, and the timbre parameter into the reconstruction network, such that the reconstruction network performs an acoustic feature reconstruction on the first speech content information, the fundamental frequency and energy parameter, and the timbre parameter by a deep recurrent neural network to acquire the third acoustic feature.

### Claim 8 (depends on 1)

The method as claimed in claim 1, further comprising:
acquiring a first speech and a second speech of the third user;
extracting second speech content information and a fourth acoustic feature from the first speech;
extracting a fifth acoustic feature from the second speech;
acquiring a reconstructed sixth acoustic feature by inputting the second speech content information, the fourth acoustic feature, and the fifth acoustic feature into a voice conversion model to be trained;
adjusting model parameters in the voice conversion model to be trained based on a difference between the sixth acoustic feature and the fourth acoustic feature, and returning to the acquiring the first speech and the second speech of the third user until the difference between the sixth acoustic feature and the fourth acoustic feature satisfies a preset training end condition; and
determining the voice conversion model to be trained after a last adjusting of model parameters as the pre-trained voice conversion model.

### Claim 9

An electronic device, comprising:
at least one processor; and
a memory communicatively connected to the at least one processor; wherein,
the memory is configured to store instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is caused to perform:
acquiring a source speech of a first user and a reference speech of a second user;
extracting first speech content information and a first acoustic feature from the source speech;
extracting a second acoustic feature from the reference speech;
acquiring a reconstructed third acoustic feature by inputting the first speech content information, the first acoustic feature, and the second acoustic feature into a pre-trained voice conversion model, in which the pre-trained voice conversion model is acquired by training based on speeches of a third user; and
synthesizing a target speech based on the third acoustic feature.

### Claim 10 (depends on 9)

The electronic device as claimed in claim 9, wherein extracting the first speech content information from the source speech comprises:
acquiring a phonetic posterior gram by inputting the source speech into a pre-trained multilingual automatic speech recognition model; and
using the phonetic posterior gram as the first speech content information.

### Claim 11 (depends on 9)

The electronic device as claimed in claim 9, wherein the first acoustic feature, the second acoustic feature, and the third acoustic feature are Mel features.

### Claim 12 (depends on 9)

The electronic device as claimed in claim 9, wherein the voice conversion model comprises a hidden-variable network, a timbre network, and a reconstruction network, and acquiring the reconstructed third acoustic feature by inputting the first speech content information, the first acoustic feature, and the second acoustic feature into the pre-trained voice conversion model comprises:
acquiring a fundamental frequency and an energy parameter by inputting the first acoustic feature into the hidden-variable network;
acquiring a timbre parameter by inputting the second acoustic feature into the timbre network; and
acquiring the third acoustic feature by inputting the first speech content information, the fundamental frequency and energy parameter, and the timbre parameter into the reconstruction network.

### Claim 13 (depends on 12)

The electronic device as claimed in claim 12, wherein acquiring the fundamental frequency and energy parameter by inputting the first acoustic feature into the hidden-variable network comprises:
inputting the first acoustic feature into the hidden-variable network, such that the hidden-variable network compresses the first acoustic feature on a frame scale, and extracts the fundamental frequency and energy parameter from the compressed first acoustic feature.

### Claim 14 (depends on 12)

The electronic device as claimed in claim 12, wherein acquiring the timbre parameter by inputting the second acoustic feature into the timbre network comprises:
inputting the second acoustic feature into the timbre network, such that the timbre network abstracts the second acoustic feature by a deep recurrent neural network and a variational auto encoder to acquire the timbre parameter.

### Claim 15 (depends on 12)

The electronic device as claimed in claim 12, wherein acquiring the third acoustic feature by inputting the first speech content information, the fundamental frequency and energy parameter, and the timbre parameter into the reconstruction network comprises:
inputting the first speech content information, the fundamental frequency and energy parameter, and the timbre parameter into the reconstruction network, such that the reconstruction network performs an acoustic feature reconstruction on the first speech content information, the fundamental frequency and energy parameter, and the timbre parameter by a deep recurrent neural network to acquire the third acoustic feature.

### Claim 16 (depends on 9)

The electronic device as claimed in claim 9, wherein when the instructions are executed by the at least one processor, the at least one processor is caused to perform:
acquiring a first speech and a second speech of the third user;
extracting second speech content information and a fourth acoustic feature from the first speech;
extracting a fifth acoustic feature from the second speech;
acquiring a reconstructed sixth acoustic feature by inputting the second speech content information, the fourth acoustic feature, and the fifth acoustic feature into a voice conversion model to be trained;
adjusting model parameters in the voice conversion model to be trained based on a difference between the sixth acoustic feature and the fourth acoustic feature, and returning to the acquiring the first speech and the second speech of the third user until the difference between the sixth acoustic feature and the fourth acoustic feature satisfies a preset training end condition; and
determining the voice conversion model to be trained after a last adjusting of model parameters as the pre-trained voice conversion model.

### Claim 17

A non-transitory computer-readable storage medium having stored therein instructions that, when executed by a computer, the computer is caused to implement a voice conversion method comprising:
acquiring a source speech of a first user and a reference speech of a second user;
extracting first speech content information and a first acoustic feature from the source speech;
extracting a second acoustic feature from the reference speech;
acquiring a reconstructed third acoustic feature by inputting the first speech content information, the first acoustic feature, and the second acoustic feature into a pre-trained voice conversion model, in which the pre-trained voice conversion model is acquired by training based on speeches of a third user; and
synthesizing a target speech based on the third acoustic feature.

### Claim 18

The non-transitory computer-readable storage medium as claimed in 17, wherein extracting the first speech content information from the source speech comprises:
acquiring a phonetic posterior gram by inputting the source speech into a pre-trained multilingual automatic speech recognition model; and
using the phonetic posterior gram as the first speech content information.

### Claim 19

The non-transitory computer-readable storage medium as claimed in 17, wherein the voice conversion model comprises a hidden-variable network, a timbre network, and a reconstruction network, and acquiring the reconstructed third acoustic feature by inputting the first speech content information, the first acoustic feature, and the second acoustic feature into the pre-trained voice conversion model comprises:
acquiring a fundamental frequency and an energy parameter by inputting the first acoustic feature into the hidden-variable network;
acquiring a timbre parameter by inputting the second acoustic feature into the timbre network; and
acquiring the third acoustic feature by inputting the first speech content information, the fundamental frequency and energy parameter, and the timbre parameter into the reconstruction network.

### Claim 20

The non-transitory computer-readable storage medium as claimed in 17, wherein the method further comprises:
acquiring a first speech and a second speech of the third user;
extracting second speech content information and a fourth acoustic feature from the first speech;
extracting a fifth acoustic feature from the second speech;
acquiring a reconstructed sixth acoustic feature by inputting the second speech content information, the fourth acoustic feature, and the fifth acoustic feature into a voice conversion model to be trained;
adjusting model parameters in the voice conversion model to be trained based on a difference between the sixth acoustic feature and the fourth acoustic feature, and returning to the acquiring the first speech and the second speech of the third user until the difference between the sixth acoustic feature and the fourth acoustic feature satisfies a preset training end condition; and
determining the voice conversion model to be trained after a last adjusting of model parameters as the pre-trained voice conversion model.
