# US 2025/0157482 A1 · Generating parallel data for real-time speech form conversion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0157482 A1
- Document ID: US20250157482A1
- Status: Published application
- Kind: A1
- Application number: 19/021,297
- Filed: 2025-01-15
- Published: 2025-05-15
- Earliest claimed: 2023-07-21
- Assignees: Krisp Technologies, Inc.
- Applicants: Krisp Technologies, Inc.
- Inventors: STEPAN SARGSYAN; ARTUR KOBELYAN; LEVON GALOYAN; KAJIK HAKOBYAN; RIMA SHAHBAZYAN; DANIEL BAGHDASARYAN; RUBEN HASRATYAN; NAIRI HAKOBYAN; HAYK ALEKSANYAN; TIGRAN TONOYAN; ARIS HOVSEPYAN
- Web page: https://patentgrep.com/patent/US20250157482A1

## CPC

- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G10L 25/30: using neural networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/096: Transfer learning
- G06N 20/00: Machine learning
- G10L 21/003: Changing voice quality, e.g. pitch or formants
- G10L 21/055: for synchronising with other signals, e.g. video signals
- G06N 3/0475: Generative networks
- G06N 3/094: Adversarial learning
- G10L 2021/0135: {Voice conversion or morphing}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Techniques are described for generating parallel data for real-time speech form conversion. In an embodiment, based at least in part on input speech data of an original form, a speech machine learning (ML) model generates parallel speech data. The parallel speech data includes the input speech data of the original form and temporally aligned output speech data of a target form different than the original form. Each frame of the input speech data temporally corresponds to the corresponding output speech frame of the target speech form and contains a same portion of the particular content. The techniques further include training a teacher machine learning model that is offline and is substantially larger than a student machine learning model for converting speech form. Transferring “knowledge” from the trained Teacher model for training the Production Student Model that performs the speech form conversion on an end-user computing device.

## Claims

### Claim 1

A computer-implemented method comprising:
training a machine learning algorithm to generate a machine learning model for converting speech from an original form to a target form, wherein the machine learning model comprises an encoder machine learning model and a decoder machine learning model, the training comprising training a decoder machine learning algorithm to generate the decoder machine learning model at least by:
applying a previously trained encoder machine learning model on training speech data in the target form to generate a training input content-based vector set of form-agnostic data;
providing the training input content-based vector set of form-agnostic data to the decoder machine learning algorithm to generate corresponding predicted speech data in the target form.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining one or more loss functions of the decoder machine learning algorithm by comparing the corresponding predicted speech data in the target form with the training speech data in the target form;
based, at least in part, on the one or more loss functions of the decoder machine learning algorithm, adjusting one or more parameters of the decoder machine learning algorithm.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
prior to the performing the training of the decoder machine learning model, training an encoder machine learning algorithm to generate the previously trained encoder machine learning model at least by:
providing training input speech data to the encoder machine learning algorithm to generate a predicted content-based vector set of form-agnostic data.

### Claim 4 (depends on 3)

The method of claim 3, wherein the training input speech data is in the original form.

### Claim 5 (depends on 3)

The method of claim 3, wherein the training input speech data is in the target form.

### Claim 6 (depends on 3)

The method of claim 3, further comprising:
determining one or more loss functions of the encoder machine learning algorithm by comparing the predicted content-based vector set of form-agnostic data with content-based data corresponding to the training input speech data;
based, at least in part, on the one or more loss functions of the encoder machine learning algorithm, adjusting one or more parameters of the previously trained encoder machine learning model.

### Claim 7 (depends on 6)

The method of claim 6, wherein the content-based data corresponding to the training input speech data comprises phoneme-based data.

### Claim 8 (depends on 6)

The method of claim 6, wherein the content-based data corresponding to the training input speech data comprises textual data.

### Claim 9 (depends on 1)

The method of claim 1, wherein the machine learning model is a previously trained machine learning model, the method further comprising:
applying the previously trained encoder machine learning model of the previously trained machine learning model to input parallel speech data of the original form;
generating, by the previously trained encoder machine learning model, a particular content-based vector set of form-agnostic data;
applying a previously trained decoder machine learning model of the previously trained machine learning model on the particular content-based vector set of form-agnostic data;
generating, by the previously trained decoder machine learning model, predicted parallel speech data in the target form, wherein the input parallel speech data of the original form and the predicted parallel speech data in the target form are temporally aligned.

### Claim 10

A system comprising one or more processors and one or more storage media storing one or more computer programs that include instructions, which, when executed by the one or more processors, cause:
training a machine learning algorithm to generate a machine learning model for converting speech from an original form to a target form, wherein the machine learning model comprises an encoder machine learning model and a decoder machine learning model, the training comprising training a decoder machine learning algorithm to generate the decoder machine learning model at least by:
applying a previously trained encoder machine learning model on training speech data in the target form to generate a training input content-based vector set of form-agnostic data;
providing the training input content-based vector set of form-agnostic data to the decoder machine learning algorithm to generate corresponding predicted speech data in the target form.

### Claim 11 (depends on 10)

The system of claim 10, wherein the one or more programs include instructions, which, when executed by the one or more processors, further cause:
determining one or more loss functions of the decoder machine learning algorithm by comparing the corresponding predicted speech data in the target form with the training speech data in the target form;
based, at least in part, on the one or more loss functions of the decoder machine learning algorithm, adjusting one or more parameters of the decoder machine learning algorithm.

### Claim 12 (depends on 10)

The system of claim 10, wherein the one or more programs include instructions, which, when executed by the one or more processors, further cause:
prior to the performing the training of the decoder machine learning model, training an encoder machine learning algorithm to generate the previously trained encoder machine learning model at least by:
providing training input speech data to the encoder machine learning algorithm to generate a predicted content-based vector set of form-agnostic data.

### Claim 13 (depends on 12)

The system of claim 12, wherein the training input speech data is in the original form.

### Claim 14 (depends on 12)

The system of claim 12, wherein the training input speech data is in the target form.

### Claim 15 (depends on 12)

The system of claim 12, wherein the one or more programs include instructions, which, when executed by the one or more processors, further cause:
determining one or more loss functions of the encoder machine learning algorithm by comparing the predicted content-based vector set of form-agnostic data with content-based data corresponding to the training input speech data;
based, at least in part, on the one or more loss functions of the encoder machine learning algorithm, adjusting one or more parameters of the previously trained encoder machine learning model.

### Claim 16 (depends on 15)

The system of claim 15, wherein the content-based data corresponding to the training input speech data comprises phoneme-based data.

### Claim 17 (depends on 15)

The system of claim 15, wherein the content-based data corresponding to the training input speech data comprises textual data.

### Claim 18 (depends on 10)

The system of claim 10, wherein the machine learning model is a previously trained machine learning model, and wherein the one or more programs include instructions, which, when executed by the one or more processors, further cause:
applying the previously trained encoder machine learning model of the previously trained machine learning model to input parallel speech data of the original form;
generating, by the previously trained encoder machine learning model, a particular content-based vector set of form-agnostic data;
applying a previously trained decoder machine learning model of the previously trained machine learning model on the particular content-based vector set of form-agnostic data;
generating, by the previously trained decoder machine learning model, predicted parallel speech data in the target form, wherein the input parallel speech data of the original form and the predicted parallel speech data in the target form are temporally aligned.

### Claim 19

One or more non-transitory computer-readable media storing a set of instructions, wherein the set of instructions includes instructions, which, when executed by one or more hardware processors, cause:
training a machine learning algorithm to generate a machine learning model for converting speech from an original form to a target form, wherein the machine learning model comprises an encoder machine learning model and a decoder machine learning model, the training comprising training a decoder machine learning algorithm to generate the decoder machine learning model at least by:
applying a previously trained encoder machine learning model on training speech data in the target form to generate a training input content-based vector set of form-agnostic data;
providing the training input content-based vector set of form-agnostic data to the decoder machine learning algorithm to generate corresponding predicted speech data in the target form.

### Claim 20 (depends on 19)

The non-transitory computer-readable media of claim 19, wherein the set of instructions includes instructions, which, when executed by one or more hardware processors, further cause:
prior to the performing the training of the decoder machine learning model, training an encoder machine learning algorithm to generate the previously trained encoder machine learning model at least by:
providing training input speech data to the encoder machine learning algorithm to generate a predicted content-based vector set of form-agnostic data.
