# US 11,908,447 B2 · Method and apparatus for synthesizing multi-speaker speech using artificial neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,447 B2
- Document ID: US11908447B2
- Status: Granted
- Kind: B2
- Application number: 17/596,037
- Filed: 2021-08-04
- Published: 2023-06-08
- Granted: 2024-02-20
- Earliest claimed: 2020-08-04
- Assignees: Iucf-Hyu (Industry-University Cooperation Foundation Hanyang University)
- Applicants: IUCF-HYU (Industry-University Cooperation Foundation Hanyang University)
- Inventors: Joon Hyuk Chang; Jae Uk Lee
- Web page: https://patentgrep.com/patent/US11908447B2

## CPC

- G10L 13/047: Architecture of speech synthesisers
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 17/04: Training, enrolment or model building
- G10L 17/06: Decision making techniques; Pattern matching strategies
- G10L 17/18: Artificial neural networks; Connectionist approaches
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

According to an aspect, method for synthesizing multi-speaker speech using an artificial neural network comprises generating and storing a speech learning model for a plurality of users by subjecting a synthetic artificial neural network of a speech synthesis model to learning, based on speech data of the plurality of users, generating speaker vectors for a new user who has not been learned and the plurality of users who have already been learned by using a speaker recognition model, determining a speaker vector having the most similar relationship with the speaker vector of the new user according to preset criteria out of the speaker vectors of the plurality of users who have already been learned, and generating and learning a speaker embedding of the new user by subjecting the synthetic artificial neural network of the speech synthesis model to learning, by using a value of a speaker embedding of a user for the determined speaker vector as an initial value and based on speaker data of the new user.

## Claims

### Claim 1

A method for synthesizing multi-speaker speech using an artificial neural network, comprising:
generating and storing a speech learning model for a plurality of users by subjecting a synthetic artificial neural network of a speech synthesis model to learning, based on speech data of the plurality of users;
generating speaker vectors for a new user who has not been learned and the plurality of users who have already been learned by using a speaker recognition model;
determining a speaker vector having a most similar relationship with the speaker vector of the new user according to preset criteria out of the speaker vectors of the plurality of users who have already been learned;
generating and learning a speaker embedding of the new user by subjecting the synthetic artificial neural network of the speech synthesis model to learning, by using a value of a speaker embedding of a user for the determined speaker vector as an initial value and based on speaker data of the new user;
wherein the generating a speech learning model for the new user comprises performing the learning of the synthetic artificial neural network of the speech synthesis model only for a preset time that prevents overfitting,
wherein the preset time comprises a range of 10 seconds to 60 seconds, and
wherein the generating speaker vectors comprises generating the speaker vectors using an artificial neural network of the speaker recognition model, by using a speech signal of the user as an input value.

### Claim 2 (depends on 1)

The method for synthesizing multi-speaker speech using the artificial neural network of claim 1, wherein the determining a speaker vector having the most similar relationship according to preset criteria comprises:
determining based on values calculated by taking inner products of the speaker vector of the new user and the speaker vectors of the plurality of users who have already been learned.

### Claim 3 (depends on 2)

The method for synthesizing multi-speaker speech using the artificial neural network of claim 2, wherein the determining a speaker vector having the most similar relationship according to preset criteria comprises:
calculating cosine similarity values based on the values of the inner products calculated, and determining a speaker vector of a user having a highest cosine similarity value as the speaker vector having the most similar relationship with the speaker vector of the new user.

### Claim 4 (depends on 1)

The method for synthesizing multi-speaker speech using the artificial neural network of claim 1, further comprising:
converting a mel-scale spectrogram calculated through the synthetic artificial neural network of the speech synthesis model into speech through a Griffin-Lim algorithm.

### Claim 5 (depends on 1)

The method for synthesizing multi-speaker speech using the artificial neural network of claim 1, wherein the synthetic artificial neural network model of the speech synthesis model comprises a Tacotron 2 algorithm.

### Claim 6

An apparatus for synthesizing multi-speaker speech using an artificial neural network, comprising:
a speech synthesizer configured to generate a speech learning model for a plurality of users by subjecting a synthetic artificial neural network of a speech synthesis model to learning, based on speech data of the plurality of users;
a storage configured to store information on the speech learning model generated;
a speaker vector generator configured to generate speaker vectors for a new user who has not been learned and the plurality of users who have already been learned by using a speaker recognition model; and
a similar-vector determinator configured to determine a speaker vector having a most similar relationship with the speaker vector of the new user according to preset criteria out of the speaker vectors of the plurality of users who have already been learned, wherein the speech synthesizer generates and learns a speaker embedding of the new user by subjecting the synthetic artificial neural network of the speech synthesis model to learning, by using a value of a speaker embedding of a user for the determined speaker vector as an initial value and based on speaker data of the new user,
wherein the speech synthesizer is configured to perform the learning of the synthetic artificial neural network of the speech synthesis model only for a preset time that prevents overfitting,
wherein the preset time comprises a range of 10 seconds to 60 seconds, and
wherein the speaker vector generator is configured to generate the speaker vectors using an artificial neural network of the speaker recognition model, by using a speech signal of the user as an input value.

### Claim 7 (depends on 6)

The apparatus for synthesizing multi-speaker speech using the artificial neural network of claim 6, wherein the similar-vector determinator determines the speaker vector having the most similar relationship with the speaker vector of the new user based on values calculated by taking inner products of the speaker vector of the new user and the speaker vectors of the plurality of users who have already been learned.

### Claim 8 (depends on 7)

The apparatus for synthesizing multi-speaker speech using the artificial neural network of claim 7, wherein the similar-vector determinator calculates cosine similarity values based on the values of the inner products calculated, and then determines a speaker vector of a user having a highest cosine similarity value as the speaker vector having the most similar relationship with the speaker vector of the new user.

### Claim 9

An apparatus for synthesizing multi-speaker speech using an artificial neural network, comprising:
a communicator configured to receive, from an external server, information on a speech learning model for a plurality of users by subjecting a synthetic artificial neural network of a speech synthesis model to learning, based on speech data of the plurality of users;
a storage configured to store information on the speech learning model generated;
a speaker vector generator configured to generate speaker vectors for a new user who has not been learned and the plurality of users who have already been learned by using a speaker recognition model; and
a similar-vector determinator configured to determine a speaker vector having a most similar relationship with the speaker vector of the new user according to preset criteria out of the speaker vectors of the plurality of users who have already been learned; and
a speech synthesizer configured to generate and learn a speaker embedding of the new user by subjecting the synthetic artificial neural network of the speech synthesis model to learning, by using a value of a speaker embedding of a user for the determined speaker vector as an initial value and based on speaker data of the new user,
wherein the speech synthesizer is configured to perform the learning of the synthetic artificial neural network of the speech synthesis model only for a preset time that prevents overfitting,
wherein the preset time comprises a range of 10 seconds to 60 seconds, and
wherein the speaker vector generator is configured to generate the speaker vectors using an artificial neural network of the speaker recognition model, by using a speech signal of the user as an input value.