# US 2023/0056128 A1 · Speech processing method and apparatus, device and computer storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0056128 A1
- Document ID: US20230056128A1
- Status: Published application
- Kind: A1
- Application number: 17/736,175
- Filed: 2022-05-04
- Published: 2023-02-23
- Earliest claimed: 2021-08-17
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science Technology Co., Ltd.
- Inventors: Liqiang ZHANG; Jiankang HOU; Tao SUN; Lei JIA
- Web page: https://patentgrep.com/patent/US20230056128A1

## CPC

- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 13/10: Prosody rules derived from text; Stress or intonation
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L 25/21: the extracted parameters being power information
- G10L 13/047: Architecture of speech synthesisers
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

The present disclosure discloses a speech processing method and apparatus, a device and a computer storage medium, and relates to speech and deep learning technologies in the field of artificial intelligence technologies. A specific implementation solution involves: acquiring a vocoder feature obtained for text; correcting a value of an unvoiced and voiced (UV) feature in the vocoder feature according to an energy feature and/or a speech spectrum feature in the vocoder feature; and providing the corrected vocoder feature for a vocoder, so as to obtain synthesized speech.

## Claims

### Claim 1

A speech processing method, comprising:
acquiring a vocoder feature obtained for text;
correcting a value of an unvoiced and voiced (UV) feature in the vocoder feature according to an energy feature and/or a speech spectrum feature in the vocoder feature; and
providing the corrected vocoder feature for a vocoder, so as to obtain synthesized speech.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the correcting a value of a UV feature in the vocoder feature according to an energy feature in the vocoder feature comprises:
determining whether an energy feature value corresponding to a frame with a value of 1 on a value change boundary of a UV feature sequence in the vocoder feature is less than 0, and if yes, correcting a UV feature value of the frame to 0; and
determining whether a ratio of an energy feature value corresponding to a frame with a value of 0 on the value change boundary of the UV feature sequence to the energy feature value corresponding to the frame with the value of 1 adjacent thereto is greater than a preset ratio threshold, and if yes, correcting a UV feature value of the frame with the value of 0 to 1.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the correcting a value of a UV feature in the vocoder feature according to a speech spectrum feature in the vocoder feature comprises:
for each frame, if a maximum value of first M dimensions of a speech spectrum feature of the frame is less than a preset first threshold, setting a UV feature value of the frame to 1; and if the maximum value of the first M dimensions of the speech spectrum feature of the frame is greater than a preset second threshold, setting the UV feature value of the frame to 0;
wherein M is a preset positive integer, and the second threshold is greater than the first threshold.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the speech spectrum feature is a mel spectrum feature, M is 20, the first threshold is 2, and the second threshold is 2.5.

### Claim 5 (depends on 1)

The method according to claim 1, prior to the correcting a value of a UV feature in the vocoder feature according to an energy feature and/or a speech spectrum feature in the vocoder feature, further comprising:
linearly interpolating the vocoder feature according to a preset interpolation multiple.

### Claim 6 (depends on 1)

The method according to claim 1, prior to the providing the corrected vocoder feature for a vocoder, further comprising:
inversely normalizing a preset type of feature sequences in the corrected vocoder feature; wherein the inversely normalizing corresponds to normalizing the preset type of feature sequences during the training of a speech synthesis model, the speech synthesis model being a source for the acquiring a vocoder feature obtained for text.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the acquiring a vocoder feature obtained for text comprises:
acquiring a prosody feature extracted from the text; and
inputting the text and the prosody feature into a speech synthesis model to obtain the vocoder feature;
wherein the speech synthesis model comprises a second encoder, a second decoder and a post-net;
the second encoder is configured to extract a language feature from the text, splice the language feature with the prosody feature, or splice the language feature, the prosody feature and a speaker feature, and output a second splicing feature obtained by splicing to the second decoder;
the second decoder is configured to predict an acoustic feature of a current frame by using a predicted acoustic feature of a previous frame and the second splicing feature and outputs it to the post-net; wherein the acoustic feature comprises a speech spectrum feature; and
the post-net is configured to predict the vocoder feature by using the acoustic feature.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the post-net predicting the vocoder feature by using the acoustic feature comprises:
causing, by the post-net, the acoustic feature to be processed by a CBHG module and then predicted by N prediction modules, prediction results constituting the vocoder feature, wherein the prediction modules comprise a bidirectional Gate Recurrent Unit (GRU) and a linear projection layer, and N is a positive integer.

### Claim 9 (depends on 7)

The method according to claim 7, wherein the acquiring a prosody feature extracted from the text comprises:
inputting the text into a prosody prediction model to obtain the prosody feature;
wherein the prosody prediction model comprises a first encoder and a first decoder;
the first encoder is configured to extract a language feature from the text and then output the language feature to the first decoder; and the first decoder is configured to predict a prosody feature of a current frame by using a predicted prosody feature of a previous frame and the language feature; or
the first decoder is configured to extract a language feature from the text, splice a broadcast style feature extracted from the text with the language feature, and input an obtained first splicing feature into the first decoder; and the first decoder is configured to predict a prosody feature of a current frame by using a predicted prosody feature of a previous frame and the first splicing feature.

### Claim 10

An electronic device, comprising:
at least one processor; and
a memory communicatively connected with the at least one processor;
wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a speech processing method, wherein the speech processing method comprises:
acquiring a vocoder feature obtained for text;
correcting a value of a UV feature in the vocoder feature according to an energy feature and/or a speech spectrum feature in the vocoder feature; and
providing the corrected vocoder feature for a vocoder, so as to obtain synthesized speech.

### Claim 11 (depends on 10)

The electronic device according to claim 10, wherein the correcting a value of a UV feature in the vocoder feature according to an energy feature in the vocoder feature comprises:
determining whether an energy feature value corresponding to a frame with a value of 1 on a value change boundary of a UV feature sequence in the vocoder feature is less than 0, and if yes, correct a UV feature value of the frame to 0; and
determining whether a ratio of an energy feature value corresponding to a frame with a value of 0 on the value change boundary of the UV feature sequence to the energy feature value corresponding to the frame with the value of 1 adjacent thereto is greater than a preset ratio threshold, and if yes, correct a UV feature value of the frame with the value of 0 to 1.

### Claim 12 (depends on 10)

The electronic device according to claim 10, wherein the correcting a value of a UV feature in the vocoder feature according to a speech spectrum feature in the vocoder feature comprises:
for each frame, if a maximum value of first M dimensions of a speech spectrum feature of the frame is less than a preset first threshold, setting a UV feature value of the frame to 1; and if the maximum value of the first M dimensions of the speech spectrum feature of the frame is greater than a preset second threshold, setting the UV feature value of the frame to 0;
wherein M is a preset positive integer, and the second threshold is greater than the first threshold.

### Claim 13 (depends on 12)

The electronic device according to claim 12, wherein the speech spectrum feature is a mel spectrum feature, M is 20, the first threshold is 2, and the second threshold is 2.5.

### Claim 14 (depends on 10)

The electronic device according to claim 10, prior to the correcting a value of a UV feature in the vocoder feature according to an energy feature and/or a speech spectrum feature in the vocoder feature, further comprising:
linearly interpolating the vocoder feature according to a preset interpolation multiple.

### Claim 15 (depends on 10)

The electronic device according to claim 10, prior to the providing the corrected vocoder feature for a vocoder, further comprising:
inversely normalizing a preset type of feature sequences in the vocoder feature corrected by the UV correction unit; wherein the inversely normalizing corresponds to normalizing the preset type of feature sequences during the training of a speech synthesis model, the speech synthesis model being a source for the acquiring a vocoder feature obtained for text.

### Claim 16 (depends on 10)

The electronic device according to claim 10, wherein the acquiring a vocoder feature obtained for text comprises:
acquiring a prosody feature extracted from the text; and inputting the text and the prosody feature into a speech synthesis model to obtain the vocoder feature;
wherein the speech synthesis model comprises a second encoder, a second decoder and a post-net;
the second encoder is configured to extract a language feature from the text, splice the language feature with the prosody feature, or splice the language feature, the prosody feature and a speaker feature, and output a second splicing feature obtained by splicing to the second decoder;
the second decoder is configured to predict an acoustic feature of a current frame by using a predicted acoustic feature of a previous frame and the second splicing feature and output it to the post-net; wherein the acoustic feature comprises a speech spectrum feature; and
the post-net is configured to predict the vocoder feature by using the acoustic feature.

### Claim 17 (depends on 16)

The electronic device according to claim 16, wherein the post-net predicting the vocoder feature by using the acoustic feature comprises:
inputting the text into a prosody prediction model to obtain the prosody feature;
wherein the prosody prediction model comprises a first encoder and a first decoder;
the first encoder is configured to extract a language feature from the text and then output the language feature to the first decoder; and the first decoder is configured to predict a prosody feature of a current frame by using a predicted prosody feature of a previous frame and the language feature; or
the first decoder is configured to extract a language feature from the text, splice a broadcast style feature extracted from the text with the language feature, and input an obtained first splicing feature into the first decoder; and the first decoder is configured to predict a prosody feature of a current frame by using a predicted prosody feature of a previous frame and the first splicing feature.

### Claim 18 (depends on 16)

The electronic device according to claim 16, wherein the post-net predicting the vocoder feature by using the acoustic feature comprises:
causing, by the post-net, the acoustic feature to be processed by a CBHG module and then predicted by N prediction modules, prediction results constituting the vocoder feature, wherein the prediction modules comprise a bidirectional Gate Recurrent Unit (GRU) and a linear projection layer, and N is a positive integer.

### Claim 19

A non-transitory computer readable storage medium with computer instructions stored thereon, wherein the computer instructions are used for causing a speech processing method, wherein the speech processing method comprises:
acquiring a vocoder feature obtained for text;
correcting a value of an unvoiced and voiced (UV) feature in the vocoder feature according to an energy feature and/or a speech spectrum feature in the vocoder feature; and
providing the corrected vocoder feature for a vocoder, so as to obtain synthesized speech.

### Claim 20 (depends on 19)

The non-transitory computer readable storage medium according to claim 19, wherein the correcting a value of a UV feature in the vocoder feature according to an energy feature in the vocoder feature comprises:
determining whether an energy feature value corresponding to a frame with a value of 1 on a value change boundary of a UV feature sequence in the vocoder feature is less than 0, and if yes, correcting a UV feature value of the frame to 0; and
determining whether a ratio of an energy feature value corresponding to a frame with a value of 0 on the value change boundary of the UV feature sequence to the energy feature value corresponding to the frame with the value of 1 adjacent thereto is greater than a preset ratio threshold, and if yes, correcting a UV feature value of the frame with the value of 0 to 1.
