# US 2026/0065892 A1 · Method and apparatus for multilingual and multi-speaker speech synthesis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0065892 A1
- Document ID: US20260065892A1
- Status: Published application
- Kind: A1
- Application number: 19/285,697
- Filed: 2025-07-30
- Published: 2026-03-05
- Earliest claimed: 2024-08-28
- Assignees: Hyundai Motor Company; Kia Corporation
- Applicants: Hyundai Motor Company; Kia Corporation
- Inventors: Chang Hwan Kim; Sung Woong Hwang
- Web page: https://patentgrep.com/patent/US20260065892A1

## CPC

- G10L 13/027: Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08)
- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G10L 13/047: Architecture of speech synthesisers
- G10L 13/086: {Detection of language}
- G10L 15/005: {Language recognition}
- G10L 15/063: {Training}
- G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for multilingual and multispeaker speech synthesis includes receiving training text and a training audio signal obtained from a speaker uttering the training text. The method further includes identifying a language identifier corresponding to the training text and a training reference audio signal obtained from the speaker uttering reference text different from the training text. The method further includes training a speech synthesis model using training samples that include the training text, the training audio signal, the language identifier, and the training reference audio signal.

## Claims

### Claim 1

A method comprising:
receiving training text and a training audio signal obtained from a speaker uttering the training text;
identifying a language identifier corresponding to the training text and a training reference audio signal obtained from the speaker uttering reference text different from the training text; and
training a speech synthesis model using training samples that include the training text, the training audio signal, the language identifier, and the training reference audio signal,
wherein training the speech synthesis model includes:
transforming, by a language embedding module of the speech synthesis model, the language identifier into a language embedding;
transforming, by a speaker encoder of the speech synthesis model, the training audio signal and the training reference audio signal into speaker embeddings; and
determining a loss based on a first speaker embedding transformed from the training reference audio signal, a second speaker embedding transformed from the training audio signal, and the language embedding; and
updating parameters of the speaker encoder using the loss.

### Claim 2 (depends on 1)

The method of claim 1, wherein the speaker encoder is trained to generate the speaker embeddings independent of language-specific characteristics of the training audio signal and the training reference audio signal.

### Claim 3 (depends on 1)

The method of claim 1 wherein the speaker encoder is trained:
to embed the training audio signal and the training reference audio signal into a feature space in which the language embedding is embedded;
to place the first speaker embedding and the second speaker embedding close to each other in the feature space; and
to place the first speaker embedding and the language embedding far apart.

### Claim 4 (depends on 1)

The method of claim 1, wherein the loss includes a triplet loss which uses the first speaker embedding, the second speaker embedding, and the language embedding as an anchor, a positive sample, and a negative sample, respectively.

### Claim 5 (depends on 1)

The method of claim 1, wherein the training text includes a sequence of one or more characters, and
wherein the language identifier includes a sequence of identifiers corresponding to individual characters within the training text.

### Claim 6 (depends on 1)

The method of claim 1, wherein training the speech synthesis model further includes:
encoding, by a posterior encoder of the speech synthesis model, a spectrogram of the training audio signal into a latent variable;
outputting, by a decoder of the speech synthesis model, a transformed latent variable based on the latent variable, the language embedding, and the first speaker embedding; and
wherein the decoder is trained to remove language-specific characteristics of the training audio signal by normalizing the latent variable based on the language embedding.

### Claim 7 (depends on 6)

The method of claim 6, wherein training the speech synthesis model further includes:
transforming, by a character embedding module of the speech synthesis model, the training text into character embeddings;
encoding, by an encoder of the speech synthesis model, the character embeddings into text feature vectors;
generating, by a projection module of the speech synthesis model, a distribution of the text feature vectors;
estimating, by an alignment data estimator of the speech synthesis model, alignment data based on the transformed latent variable and the distribution of the text feature vectors; and
predicting, by a stochastic duration predictor of the speech synthesis model, a duration associated with speech characteristics of the speaker for each phoneme of the training text based on the first speaker embedding, the text feature vectors, and the alignment data.

### Claim 8 (depends on 1)

The method of claim 1, wherein identifying the language identifier includes:
identifying, from among a plurality of audio signals obtained from the speaker, one or more audio signals other than the training audio signal; and
randomly selecting the training reference audio signal among the one or more audio signals.

### Claim 9 (depends on 1)

The method of claim 1, wherein training the speech synthesis model further includes:
providing the language embedding and the first speaker embedding as input to sub-networks subsequent to the language embedding module and the speaker encoder to generate a synthesized audio signal corresponding to the training text,
wherein the second speaker embedding is not used for generating the synthesized audio signal.

### Claim 10

An apparatus comprising:
a memory configured to store instructions;
at least one processor configured, by executing the instructions, to:
receive training text and a training audio signal obtained from a predetermined speaker uttering the training text;
identify a language identifier corresponding to the training text and a training reference audio signal obtained from the speaker uttering reference text different from the training text; and
train a speech synthesis model by using training samples that include the training text, the training audio signal, the language identifier, and the training reference audio signal,
wherein the speech synthesis model comprises:
a language embedding module configured to transform the language identifier into a language embedding; and
a speaker encoder configured to transform the training audio signal and the training reference audio signal into speaker embeddings, and
wherein the at least one processor is further configured to:
determine a loss based on a first speaker embedding transformed from the training reference audio signal, a second speaker embedding transformed from the training audio signal, and the language embedding; and
update parameters of the speaker encoder using the loss.

### Claim 11

A method comprising:
receiving a speech synthesis request including input text and information on a target speaker;
identifying a language identifier corresponding to the input text and a reference audio signal obtained from the target speaker uttering reference text different from the input text; and
generating, by applying the input text, the language identifier, and the reference audio signal to a speech synthesis model, a synthesized audio signal that simulates speech by the target speaker uttering the input text;
wherein generating the synthesized audio signal includes:
transforming, by a language embedding module of the speech synthesis model, the language identifier into a target language embedding; and
transforming, by a speaker encoder of the speech synthesis model, the reference audio signal into a target speaker embedding, and
wherein the speaker encoder has been trained by using a loss determined based on speaker embeddings transformed from a plurality of training audio signals from a speaker and a language embedding transformed from training text corresponding to one of the plurality of training audio signals.

### Claim 12 (depends on 11)

The method of claim 11, wherein the speaker encoder has been trained to generate the speaker embeddings independent of language-specific characteristics of the training audio signals.

### Claim 13 (depends on 11)

The method of claim 11, wherein the plurality of training audio signals includes a first training audio signal corresponding to reference text different from the training text and a second training audio signal corresponding to the training text,
wherein the speaker encoder has been trained:
to embed the first training audio signal and the second training audio signal into a feature space in which the language embedding transformed from the training text is embedded;
to place a first speaker embedding transformed from the first training audio signal and a second speaker embedding transformed from the second training audio signal close to each other in the feature space; and
to place the first speaker embedding and the language embedding far apart.

### Claim 14 (depends on 13)

The method of claim 13, wherein the loss includes a triplet loss, which uses the first speaker embedding, the second speaker embedding, and the language embedding for the training text as an anchor, a positive sample, and a negative sample, respectively.

### Claim 15 (depends on 11)

The method of claim 11, wherein the input text includes a sequence of one or more characters, and the language identifier includes a sequence of identifiers corresponding to individual characters within the input text.

### Claim 16 (depends on 11)

The method of claim 11, wherein generating the synthesized audio signal further includes:
outputting, by an inverted decoder of the speech synthesis model, a transformed latent variable based on a latent variable derived from the input text and the reference audio signal, the target language embedding, and the target speaker embedding; and
generating, by an audio generator of the speech synthesis model, the synthesized audio signal from the transformed latent variable.

### Claim 17 (depends on 16)

The method of claim 16, wherein outputting the transformed latent variable includes:
conditioning, by the inverted decoder, the latent variable on the target speaker embedding and the target language embedding; and
outputting, by the inverted decoder, the transformed latent variable based on the conditioned latent variable.

### Claim 18 (depends on 16)

The method of claim 16, wherein generating the synthesized audio signal from the transformed latent variable includes:
conditioning, by the audio generator, the transformed latent variable on the target speaker embedding, and
generating, by the audio generator, the synthesized audio signal from the conditioned transformed latent variable.

### Claim 19 (depends on 16)

The method of claim 16, wherein generating the synthesized audio signal further includes:
transforming, by a character embedding module of the speech synthesis model, the input text into character embeddings;
encoding, by an encoder of the speech synthesis model, the character embeddings into text feature vectors;
generating, by a projection module of the speech synthesis model, a distribution of the text feature vectors;
predicting, by a stochastic duration predictor of the speech synthesis model, a duration associated with speech characteristics of the target speaker for each phoneme of the input text, based on the target speaker embedding and the text feature vectors; and
generating, by an alignment unit of the speech synthesis model, the latent variable based on the distribution of the text feature vectors and the duration predicted for each phoneme.

### Claim 20 (depends on 11)

An apparatus comprising:
a memory configured to store instructions; and
at least one processor configured, when executing the instructions, to perform the method of claim 11.
