US 2025/0029629 A1Published application

Pre-trained machine learning models for real-time speech form conversion

AssigneeKrisp Technologies, Inc.

InventorsSTEPAN SARGSYAN, ARTUR KOBELYAN, LEVON GALOYAN, KAJIK HAKOBYAN, RIMA SHAHBAZYAN, DANIEL BAGHDASARYAN, RUBEN HASRATYAN, NAIRI HAKOBYAN, HAYK ALEKSANYAN, TIGRAN TONOYAN, ARIS HOVSEPYAN

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Techniques are described for generating parallel data for real-time speech form conversion. In an embodiment, based at least in part on input speech data of an original form, a speech machine learning (ML) model generates parallel speech data. The parallel speech data includes the input speech data of the original form and temporally aligned output speech data of a target form different than the original form. Each frame of the input speech data temporally corresponds to the corresponding output speech frame of the target speech form and contains a same portion of the particular content. The techniques further include training a teacher machine learning model that is offline and is substantially larger than a student machine learning model for converting speech form. Transferring “knowledge” from the trained Teacher model for training the Production Student Model that performs the speech form conversion on an end-user computing device.

Claims22

Dependent claims

A computer-implemented method comprising:

training a first machine learning algorithm to generate a first machine learning model including a first plurality of parameters of the first machine learning model, at least by:
providing first training input speech data in an original form to the first machine learning model to generate a first predicted speech-related data set in a particular form;
based at least in part on the first training input speech data, generating the first predicted speech-related data set in the particular form by the first machine learning model;
wherein each frame of the first predicted speech-related data set is based at least in part, on a) a corresponding frame in the first training input speech data and b) a plurality of frames that are temporally subsequent to the corresponding frame in the first training input speech data;
training a second machine learning algorithm to generate a second machine learning model, at least by:
providing second training input speech data in the original form to the second machine learning model to generate a second predicted speech-related data set in the particular form;
based at least in part on the second training input speech data, generating the second predicted speech-related data set in the particular form by the second machine learning model;
determining one or more loss functions of the second machine learning model at least by:
a) comparing the second predicted speech-related data set in the particular form with target speech-related data set in the particular form corresponding to the second training input speech data in the original form, and
b) comparing a subset of a second plurality of parameters of the second machine learning model with a corresponding subset of the first plurality of parameters of the first machine learning model;
wherein the first machine learning model is trained to generate a particular speech-related data set in the particular form from particular input speech data in the original form;
based on the one or more loss functions, adjusting the second plurality of parameters of the second machine learning algorithm that includes the subset of the second plurality of parameters.

9 more elements

A system comprising one or more processors and one or more storage media storing one or more computer programs that include instructions, which, when executed by the one or more processors, cause:

training a first machine learning algorithm to generate a first machine learning model including a first plurality of parameters of the first machine learning model, at least by:
providing first training input speech data in an original form to the first machine learning model to generate a first predicted speech-related data set in a particular form;
based at least in part on the first training input speech data, generating the first predicted speech-related data set in the particular form by the first machine learning model;
wherein each frame of the first predicted speech-related data set is based at least in part, on a) a corresponding frame in the first training input speech data and b) a plurality of frames that are temporally subsequent to the corresponding frame in the first training input speech data;
training a second machine learning algorithm to generate a second machine learning model, at least by:
providing second training input speech data in the original form to the second machine learning model to generate a second predicted speech-related data set in the particular form;
based at least in part on the second training input speech data, generating the second predicted speech-related data set in the particular form by the second machine learning model;
determining one or more loss functions of the second machine learning model at least by:
a) comparing the second predicted speech-related data set in the particular form with target speech-related data set in the particular form corresponding to the second training input speech data in the original form, and
b) comparing a subset of a second plurality of parameters of the second machine learning model with a corresponding subset of the first plurality of parameters of the first machine learning model;
wherein the first machine learning model is trained to generate a particular speech-related data set in the particular form from particular input speech data in the original form;
based on the one or more loss functions, adjusting the second plurality of parameters of the second machine learning algorithm that includes the subset of the second plurality of parameters.

9 more elements

One or more non-transitory computer-readable media storing a set of instructions, wherein the set of instructions includes instructions, which, when executed by one or more hardware processors, cause:

training a first machine learning algorithm to generate a first machine learning model including a first plurality of parameters of the first machine learning model, at least by:
providing first training input speech data in an original form to the first machine learning model to generate a first predicted speech-related data set in a particular form;
based at least in part on the first training input speech data, generating the first predicted speech-related data set in the particular form by the first machine learning model;
wherein each frame of the first predicted speech-related data set is based at least in part, on a) a corresponding frame in the first training input speech data and b) a plurality of frames that are temporally subsequent to the corresponding frame in the first training input speech data;
training a second machine learning algorithm to generate a second machine learning model, at least by:
providing second training input speech data in the original form to the second machine learning model to generate a second predicted speech-related data set in the particular form;
based at least in part on the second training input speech data, generating the second predicted speech-related data set in the particular form by the second machine learning model;
determining one or more loss functions of the second machine learning model at least by:
a) comparing the second predicted speech-related data set in the particular form with target speech-related data set in the particular form corresponding to the second training input speech data in the original form, and
b) comparing a subset of a second plurality of parameters of the second machine learning model with a corresponding subset of the first plurality of parameters of the first machine learning model;
wherein the first machine learning model is trained to generate a particular speech-related data set in the particular form from particular input speech data in the original form;
based on the one or more loss functions, adjusting the second plurality of parameters of the second machine learning algorithm that includes the subset of the second plurality of parameters.

9 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/775,232Filed 17 Jul 2024
Priority: 21 Jul 2023
Published: 23 Jan 2025
Term ends: Not estimatedThe source records don’t support an estimate.

Classification12 codes

G06N 3/0442
Recurrent networks, e.g. Hopfield networks: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
G10L 25/30
Characterised by the analysis technique: using neural networks
G06N 3/0455
Combinations of networks: auto-encoder networks; Encoder-decoder networks
G06N 3/096
Learning methods: transfer learning