# US 2026/0065896 A1 · Method and apparatus for speech synthesis for multilignual and multispeaker

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0065896 A1
- Document ID: US20260065896A1
- Status: Published application
- Kind: A1
- Application number: 19/190,349
- Filed: 2025-04-25
- Published: 2026-03-05
- Earliest claimed: 2024-08-28
- Assignees: Hyundai Motor Company; Kia Corporation
- Applicants: Hyundai Motor Company; Kia Corporation
- Inventors: Chang Hwan Kim
- Web page: https://patentgrep.com/patent/US20260065896A1

## CPC

- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 13/10: Prosody rules derived from text; Stress or intonation
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G10L 13/047: Architecture of speech synthesisers
- G10L 13/086: {Detection of language}
- G10L 25/30: using neural networks
- G10L 2013/105: {Duration}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A speech synthesis apparatus includes a memory configured to store language information configured by a user and audio samples of a speaker corresponding to speaker information selected by the user. The speech synthesis apparatus also includes a processor configured to generate an audio signal corresponding to input text by applying a speech synthesis model to the input text, the language information, and the audio samples in response to a speech synthesis request of the user. The speech synthesis model is trained to generate an audio signal including features of training text and features of a training audio signal. Language information of the training text and speaker information of the training audio signal are removed from the generated audio signal.

## Claims

### Claim 1

A speech synthesis apparatus comprising:
a memory configured to store language information configured by a user and audio samples of a speaker corresponding to speaker information selected by the user; and
a processor configured to generate an audio signal corresponding to input text by applying a speech synthesis model to the input text, the language information, and the audio samples in response to a speech synthesis request of the user,
wherein the speech synthesis model is trained to generate an audio signal including features of training text and features of a training audio signal, and wherein language information of the training text and speaker information of the training audio signal are removed from the generated audio signal.

### Claim 2 (depends on 1)

The speech synthesis apparatus of claim 1, wherein, when the input text includes characters corresponding to multiple languages, the language information is configured for each character.

### Claim 3 (depends on 1)

The speech synthesis apparatus of claim 1, wherein the speech synthesis model is trained to remove the language information of the training text and the speaker information of the training audio signal by normalizing a training latent variable including the features of the training text and the features of the training audio signal based on the language information of the training text and the speaker information of the training audio signal.

### Claim 4 (depends on 1)

The speech synthesis apparatus of claim 1, wherein, when generating duration of each phoneme of the training text, the speech synthesis model is configured to utilize the training text, a language embedding of the training text, and a speaker embedding of the training audio signal, and wherein the speech synthesis model is trained to utilize language information inherent in the training text and the language embedding of the training text and exclude language information inherent in the speaker embedding of the training audio signal.

### Claim 5 (depends on 1)

The speech synthesis apparatus of claim 1, wherein the speech synthesis model includes:
a language embedding module configured to transform the language information to a language embedding;
a character embedding module configured to transform the input text into character embeddings;
an encoder configured to encode the character embeddings to text feature vectors;
a speaker encoder configured to encode the audio samples to output a speaker embedding;
a duration predictor configured to predict phoneme duration data including duration of each phoneme of the input text based on the text feature vectors, the language embedding, and the speaker embedding;
a projection module configured to generate a distribution of the text feature vectors;
an alignment unit configured to generate a latent variable based on the distribution of the text feature vectors and the phoneme duration data;
an inverted decoder configured to output a transformed latent variable based on the latent variable, the speaker embedding, and the language embedding; and
an audio generator configured to generate the audio signal from the transformed latent variable.

### Claim 6 (depends on 5)

The speech synthesis apparatus of claim 5, wherein the inverted decoder is configured to de-normalizes the latent variable based on the speaker embedding and the language embedding and outputs the transformed latent variable based on the de-normalized latent variable.

### Claim 7 (depends on 5)

The speech synthesis apparatus of claim 5, wherein, when generating duration of each phoneme of the input text, the duration predictor is configured to:
utilize the language embedding and the speaker embedding; and
utilize language information inherent in the input text and the language embedding and exclude language information inherent in the speaker embedding.

### Claim 8

A speech synthesis method performed by a speech synthesis apparatus, the method comprising:
receiving a speech synthesis request for input text, wherein the speech synthesis request includes language information and speaker information configured by a user; and
generating an audio signal corresponding to the input text by applying a speech synthesis model to the input text, the language information, and audio samples of a speaker corresponding to the speaker information,
wherein the speech synthesis model is trained to generate an audio signal including features of training text and features of a training audio signal, and wherein language information of the training text and speaker information of the training audio signal are removed from the generated audio signal.

### Claim 9 (depends on 8)

The method of claim 8, wherein, when the input text includes characters corresponding to multiple languages, the language information is configured for each character.

### Claim 10 (depends on 8)

The method of claim 8, wherein the speech synthesis model is trained to remove the language information of the training text and the speaker information of the training audio signal by normalizing a training latent variable including the features of the training text and the features of the training audio signal based on the language information of the training text and the speaker information of the training audio signal.

### Claim 11 (depends on 8)

The method of claim 8, wherein, generating duration of each phoneme of the training text includes utilizing the training text, language embedding of the training text, and speaker embedding of the training audio signal, and the speech synthesis model is trained to utilize language information inherent in the training text and the language embedding of the training text and exclude language information inherent in the speaker embedding of the training audio signal.

### Claim 12 (depends on 8)

The method of claim 8, wherein generating the audio signal corresponding to the input text by applying the speech synthesis model to the input text includes:
transforming, by a language embedding module, the language information to a language embedding;
transforming, by a character embedding module, the input text into character embeddings;
encoding, by an encoder, the character embeddings to text feature vectors;
encoding, by a speaker encoder, the audio samples to output a speaker embedding;
predicting, by a duration predictor, phoneme duration data including duration of each phoneme of the input text based on the text feature vectors, the language embedding, and the speaker embedding;
generating, by a projection module, a distribution of the text feature vectors;
generating, by an alignment unit, a latent variable based on the distribution of the text feature vectors and the phoneme duration data;
outputting, by an inverted decoder, a transformed latent variable based on the latent variable, the speaker embedding, and the language embedding; and
generating, by an audio generator, the audio signal from the transformed latent variable.

### Claim 13 (depends on 12)

The method of claim 12, wherein outputting the transformed latent variable based on the latent variable includes:
de-normalizing the latent variable based on the speaker embedding and the language embedding; and
outputting the transformed latent variable based on the de-normalized latent variable.

### Claim 14 (depends on 12)

The method of claim 12, wherein, generating duration of each phoneme of the input text includes:
utilizing the language embedding and the speaker embedding; and
utilizing language information inherent in the input text and the language embedding and excludes language information inherent in the speaker embedding.
