# US 2023/0410791 A1 · Text-to-speech synthesis method, electronic device, and computer-readable storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410791 A1
- Document ID: US20230410791A1
- Status: Published application
- Kind: A1
- Application number: 18/212,140
- Filed: 2023-06-20
- Published: 2023-12-21
- Earliest claimed: 2022-06-21
- Assignees: Ubtech Robotics Corp Ltd
- Applicants: Ubtech Robotics Corp Ltd
- Inventors: Wan Ding; Dongyuan Huang; Zehong Zheng; Linhuang Yan; Zhiyong Yang
- Web page: https://patentgrep.com/patent/US20230410791A1

## CPC

- G10L 13/10: Prosody rules derived from text; Stress or intonation
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A text-to-speech synthesis method, an electronic device, and a computer-readable storage medium are provided. The method includes: obtaining prosodic pause features of an input text by performing a prosodic pause prediction processing on the input text, and dividing the input text into a plurality of prosodic phrases according to the prosodic pause features; synthesizing short sentence audios according to the prosodic phrases by performing a streamed speech synthesis processing on each of the prosodic phrases in the input text in a manner of asynchronous processing of a thread pool; and performing an audio playback operation of the input text according to the short sentence audios corresponding to the first prosodic phrase of the input text, in response to synthesizing the short sentence audio corresponding to the first prosodic phrase of the input text.

## Claims

### Claim 1

A computer-implemented text-to-speech synthesis method for an electronic device, comprising:
obtaining prosodic pause features of a text input into the electronic device by performing a prosodic pause prediction processing on the input text, and dividing the input text into a plurality of prosodic phrases according to the prosodic pause features;
synthesizing short sentence audios according to the prosodic phrases by performing a streamed speech synthesis processing on each of the prosodic phrases in the input text in a manner of asynchronous processing of a thread pool, wherein the thread pool includes a text normalization processing thread, a phoneme prediction processing thread, a prosody prediction processing thread, and a speech synthesis processing thread; and
performing an audio playback operation of the input text according to the short sentence audios corresponding to the first prosodic phrase of the input text, in response to synthesizing the short sentence audio corresponding to the first prosodic phrase of the input text.

### Claim 2 (depends on 1)

The method of claim 1, wherein the synthesizing the short sentence audios according to the prosodic phrases by performing the streamed speech synthesis processing on each of the prosodic phrases in the input text in the manner of asynchronous processing of the thread pool comprises:
taking each of the prosodic phrases as a target prosodic phrase to transmit in an order of a position of the prosodic phrase in the input text to the rhythm prediction processing thread for processing;
obtaining a prosody characteristic corresponding to the target prosodic phrase by performing a prosody prediction processing on the target prosodic phrase after the prosody prediction processing thread receives the target prosodic phrase, and transmitting the target prosodic phrase to the phoneme prediction processing thread after obtaining the prosody characteristic corresponding to the target prosodic phrase;
obtaining a phoneme feature of the target prosodic phrase by performing a phoneme prediction processing on the target prosodic phrase after the phoneme prediction processing thread receives the target prosodic phrase, and transmitting the target prosodic phrase to the phoneme duration prediction processing thread after obtaining the phoneme feature corresponding to target prosodic phrase;
obtaining the phoneme duration feature of the target prosodic phrase by performing a phoneme duration prediction processing on the target prosodic phrase after the phoneme duration prediction processing thread receives the target prosodic phrase, and transmitting the target prosodic phrase to the speech synthesis processing thread after obtaining the phoneme duration feature corresponding to target prosodic phrase; and
synthesizing the short sentence audio corresponding to the target prosodic phrase according to the target prosodic phrase and the prosody characteristic, phoneme feature, and phoneme duration feature corresponding to the target prosodic phrase after the speech synthesis processing thread receives the target prosodic phrase.

### Claim 3 (depends on 2)

The method of claim 2, wherein the taking each of the prosodic phrases as the target prosodic phrase to transmit in the order of the position of the prosodic phrase in the input text to the rhythm prediction processing thread for processing comprises:
performing an index numbering processing on each of the prosodic phrases in the order of the position of the prosodic phrase in the input text so that each of the prosodic phrases corresponds to a unique index number; and
taking each of the prosodic phrases as the target prosodic phrase to transmit in the order of the index number to the rhythm prediction processing thread for processing, and stopping transmitting the target prosodic phrase to the rhythm prediction processing thread in response to the index number being determined as a maximum number.

### Claim 4 (depends on 1)

The method of claim 1, wherein the synthesizing the short sentence audios according to the prosodic phrases by performing the streamed speech synthesis processing on each of the prosodic phrases in the input text in the manner of asynchronous processing of the thread pool further comprises:
obtaining a thread queue for performing the streamed speech synthesis processing on each of the prosodic phrases in the input text by connecting the text normalization processing thread, the phoneme prediction processing thread, the prosody prediction processing thread, and the speech synthesis processing thread in the thread pool in a sequential manner.

### Claim 5

An electronic device, comprising:
a processor;
a memory coupled to the processor; and
one or more computer programs stored in the memory and executable on the processor;
wherein, the one or more computer programs comprise:
instructions for obtaining prosodic pause features of a text input into the electronic device by performing a prosodic pause prediction processing on the input text, and dividing the input text into a plurality of prosodic phrases according to the prosodic pause features;
instructions for synthesizing short sentence audios according to the prosodic phrases by performing a streamed speech synthesis processing on each of the prosodic phrases in the input text in a manner of asynchronous processing of a thread pool, wherein the thread pool includes a text normalization processing thread, a phoneme prediction processing thread, a prosody prediction processing thread, and a speech synthesis processing thread; and
instructions for performing an audio playback operation of the input text according to the short sentence audios corresponding to the first prosodic phrase of the input text, in response to synthesizing the short sentence audio corresponding to the first prosodic phrase of the input text.

### Claim 6 (depends on 5)

The electronic device of claim 5, wherein the synthesizing the short sentence audios according to the prosodic phrases by performing the streamed speech synthesis processing on each of the prosodic phrases in the input text in the manner of asynchronous processing of the thread pool comprises:
taking each of the prosodic phrases as a target prosodic phrase to transmit in an order of a position of the prosodic phrase in the input text to the rhythm prediction processing thread for processing;
obtaining a prosody characteristic corresponding to the target prosodic phrase by performing a prosody prediction processing on the target prosodic phrase after the prosody prediction processing thread receives the target prosodic phrase, and transmitting the target prosodic phrase to the phoneme prediction processing thread after obtaining the prosody characteristic corresponding to the target prosodic phrase;
obtaining a phoneme feature of the target prosodic phrase by performing a phoneme prediction processing on the target prosodic phrase after the phoneme prediction processing thread receives the target prosodic phrase, and transmitting the target prosodic phrase to the phoneme duration prediction processing thread after obtaining the phoneme feature corresponding to target prosodic phrase;
obtaining the phoneme duration feature of the target prosodic phrase by performing a phoneme duration prediction processing on the target prosodic phrase after the phoneme duration prediction processing thread receives the target prosodic phrase, and transmitting the target prosodic phrase to the speech synthesis processing thread after obtaining the phoneme duration feature corresponding to target prosodic phrase; and
synthesizing the short sentence audio corresponding to the target prosodic phrase according to the target prosodic phrase and the prosody characteristic, phoneme feature, and phoneme duration feature corresponding to the target prosodic phrase after the speech synthesis processing thread receives the target prosodic phrase.

### Claim 7 (depends on 6)

The electronic device of claim 6, wherein the taking each of the prosodic phrases as the target prosodic phrase to transmit in the order of the position of the prosodic phrase in the input text to the rhythm prediction processing thread for processing comprises:
performing an index numbering processing on each of the prosodic phrases in the order of the position of the prosodic phrase in the input text so that each of the prosodic phrases corresponds to a unique index number; and
taking each of the prosodic phrases as the target prosodic phrase to transmit in the order of the index number to the rhythm prediction processing thread for processing, and stopping transmitting the target prosodic phrase to the rhythm prediction processing thread in response to the index number being determined as a maximum number.

### Claim 8 (depends on 5)

The electronic device of claim 5, wherein the synthesizing the short sentence audios according to the prosodic phrases by performing the streamed speech synthesis processing on each of the prosodic phrases in the input text in the manner of asynchronous processing of the thread pool further comprises:
obtaining a thread queue for performing the streamed speech synthesis processing on each of the prosodic phrases in the input text by connecting the text normalization processing thread, the phoneme prediction processing thread, the prosody prediction processing thread, and the speech synthesis processing thread in the thread pool in a sequential manner.

### Claim 9

A non-transitory computer-readable storage medium for storing one or more computer programs, wherein the one or more computer programs comprise:
instructions for obtaining prosodic pause features of a text input into the electronic device by performing a prosodic pause prediction processing on the input text, and dividing the input text into a plurality of prosodic phrases according to the prosodic pause features;
instructions for synthesizing short sentence audios according to the prosodic phrases by performing a streamed speech synthesis processing on each of the prosodic phrases in the input text in a manner of asynchronous processing of a thread pool, wherein the thread pool includes a text normalization processing thread, a phoneme prediction processing thread, a prosody prediction processing thread, and a speech synthesis processing thread; and
instructions for performing an audio playback operation of the input text according to the short sentence audios corresponding to the first prosodic phrase of the input text, in response to synthesizing the short sentence audio corresponding to the first prosodic phrase of the input text.

### Claim 10 (depends on 9)

The storage medium of claim 9, wherein the synthesizing the short sentence audios according to the prosodic phrases by performing the streamed speech synthesis processing on each of the prosodic phrases in the input text in the manner of asynchronous processing of the thread pool comprises:
taking each of the prosodic phrases as a target prosodic phrase to transmit in an order of a position of the prosodic phrase in the input text to the rhythm prediction processing thread for processing;
obtaining a prosody characteristic corresponding to the target prosodic phrase by performing a prosody prediction processing on the target prosodic phrase after the prosody prediction processing thread receives the target prosodic phrase, and transmitting the target prosodic phrase to the phoneme prediction processing thread after obtaining the prosody characteristic corresponding to the target prosodic phrase;
obtaining a phoneme feature of the target prosodic phrase by performing a phoneme prediction processing on the target prosodic phrase after the phoneme prediction processing thread receives the target prosodic phrase, and transmitting the target prosodic phrase to the phoneme duration prediction processing thread after obtaining the phoneme feature corresponding to target prosodic phrase;
obtaining the phoneme duration feature of the target prosodic phrase by performing a phoneme duration prediction processing on the target prosodic phrase after the phoneme duration prediction processing thread receives the target prosodic phrase, and transmitting the target prosodic phrase to the speech synthesis processing thread after obtaining the phoneme duration feature corresponding to target prosodic phrase; and
synthesizing the short sentence audio corresponding to the target prosodic phrase according to the target prosodic phrase and the prosody characteristic, phoneme feature, and phoneme duration feature corresponding to the target prosodic phrase after the speech synthesis processing thread receives the target prosodic phrase.

### Claim 11 (depends on 10)

The storage medium of claim 10, wherein the taking each of the prosodic phrases as the target prosodic phrase to transmit in the order of the position of the prosodic phrase in the input text to the rhythm prediction processing thread for processing comprises:
performing an index numbering processing on each of the prosodic phrases in the order of the position of the prosodic phrase in the input text so that each of the prosodic phrases corresponds to a unique index number; and
taking each of the prosodic phrases as the target prosodic phrase to transmit in the order of the index number to the rhythm prediction processing thread for processing, and stopping transmitting the target prosodic phrase to the rhythm prediction processing thread in response to the index number being determined as a maximum number.

### Claim 12 (depends on 9)

The storage medium of claim 9, wherein the synthesizing the short sentence audios according to the prosodic phrases by performing the streamed speech synthesis processing on each of the prosodic phrases in the input text in the manner of asynchronous processing of the thread pool further comprises:
obtaining a thread queue for performing the streamed speech synthesis processing on each of the prosodic phrases in the input text by connecting the text normalization processing thread, the phoneme prediction processing thread, the prosody prediction processing thread, and the speech synthesis processing thread in the thread pool in a sequential manner.
