# US 12,217,745 B2 · Unified speech representation learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,745 B2
- Document ID: US12217745B2
- Status: Granted
- Kind: B2
- Application number: 18/217,888
- Filed: 2023-07-03
- Published: 2023-11-16
- Granted: 2025-02-04
- Earliest claimed: 2021-05-14
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Yao Qian; Yu Wu; Kenichi Kumatani; Shujie Liu; Furu Wei; Nanshan Zeng; Xuedong David Huang; Chengyi Wang
- Web page: https://patentgrep.com/patent/US12217745B2

## CPC

- G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/096: Transfer learning
- G06N 20/00: Machine learning
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/063: {Training}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system obtains a first training data set comprising labeled speech data or both labeled and unlabeled data corresponding to a high-resource data set as well as latent speech representations based on the first training data set. The system trains a machine learning model on the first training data set to learn phonetically aware speech representations corresponding to the first training data set. The system applies the latent speech representations to a transformer context network to generate contextual representations. The system aligns each of the contextual representations with a phoneme label to generate phonetically-aware contextual representations. The system causes a refinement engine to further refine the machine learning model.

## Claims

### Claim 1

A computing system comprising:
one or more processors; and
one or more computer-readable instructions that are executable by the one or more processors to configure the computing system to at least:
obtain a first training data set comprising labeled speech data or both labeled and unlabeled data corresponding to a high-resource data set, as well as latent speech representations based on the first training data set;
train a machine learning model on the first training data set to learn phonetically aware speech representations corresponding to the first training data set;
apply the latent speech representations from the machine learning model to a transformer context network to generate contextual representations;
align each contextual representation included in the contextual representations with a phoneme label to generate phonetically-aware contextual representations; and
cause a refinement engine to further refine the machine learning model based on a refinement dataset, wherein the refinement engine fine-tunes the machine learning model on a limited labeled dataset corresponding to a low-resource target language and/or target domain; and
transform at least some of the contextual representations by randomly replacing a sub-set of the contextual representations with quantized latent speech representations.

### Claim 2 (depends on 1)

The computing system of claim 1, the one or more computer-readable instructions being further executable by the one or more processors to further configure the computing system to:
before refining the machine learning model on the refinement dataset, which operates to adapt the machine learning model to the target domain, freeze or unfreeze a feature extractor of the machine learning model based on a data size of the refinement dataset.

### Claim 3 (depends on 1)

The computing system of claim 1, the one or more computer-readable instructions being further executable by the one or more processors to further configure the computing system to:
obtain a set of contextual representations based on the first training data set;
train the machine learning model on the first training data set to align the set of contextual representations to a set of phoneme labels such that each contextual representation corresponds to a phoneme label;
obtain a set of latent representations based on the first training data set; and
train the machine learning model on the first training data set to align the set of latent representations to the set of contextual representations.

### Claim 4 (depends on 1)

The computing system of claim 1, the one or more computer-readable instructions being further executable by the one or more processors to further configure the computing system to train the machine learning model on the first training data set to align a set of contextual representations to a set of phoneme labels such that each contextual representation corresponds to the phoneme label by performing connectionist temporal classification loss at a sequence level.

### Claim 5 (depends on 1)

The computing system of claim 1, the one or more computer-readable instructions being further executable by the one or more processors to further configure the computing system to train the machine learning model on the first training data set to identify a sub-set of correct quantized latent speech representation in a set of distractors by performing contrastive loss between a set of negative latent representations, a set of positive latent representations, and a set of masked contextual representations.

### Claim 6 (depends on 1)

The computing system of claim 1, the one or more computer-readable instructions being further executable by the one or more processors to further configure the computing system to:
access a codebook comprising a set of codewords for discrete latent representations; and
align each codeword included the set of codewords to a phoneme label included in a set of phoneme labels.

### Claim 7 (depends on 6)

The computing system of claim 6, the one or more computer-readable instructions being further executable by the one or more processors to further configure the computing system to:
calculate a probability that a particular codeword corresponds to a particular phoneme label; and
aligns the particular codeword to the particular phoneme label when the probability meets or exceeds a pre-defined threshold.

### Claim 8

A method comprising:
obtaining a first training data set comprising labeled speech data or both labeled and unlabeled data corresponding to a high-resource data set, as well as latent speech representations based on the first training data set;
training a machine learning model on the first training data set to learn phonetically aware speech representations corresponding to the first training data set;
applying the latent speech representations from the machine learning model to a transformer context network to generate contextual representations;
aligning each contextual representation included in the contextual representations with a phoneme label to generate phonetically-aware contextual representations;
causing a refinement engine to further refine the machine learning model based on a refinement dataset, wherein the refinement engine fine-tunes the machine learning model on a limited labeled dataset corresponding to a low-resource target language and/or target domain; and
transforming at least some of the contextual representations by randomly replacing a sub-set of the contextual representations with quantized latent speech representations.

### Claim 9 (depends on 8)

The method of claim 8, the method further comprising:
before refining the machine learning model on the refinement dataset, which operates to adapt the machine learning model to the target domain, freezing or unfreezing a feature extractor of the machine learning model based on a data size of the refinement dataset.

### Claim 10 (depends on 8)

The method of claim 8, the method further comprising:
obtaining a set of contextual representations based on the first training data set;
training the machine learning model on the first training data set to align the set of contextual representations to a set of phoneme labels such that each contextual representation corresponds to a phoneme label;
obtaining a set of latent representations based on the first training data set; and
training the machine learning model on the first training data set to align the set of latent representations to the set of contextual representations.

### Claim 11 (depends on 8)

The method of claim 8, the method further comprising training the machine learning model on the first training data set to align a set of contextual representations to a set of phoneme labels such that each contextual representation corresponds to the phoneme label by performing connectionist temporal classification loss at a sequence level.

### Claim 12 (depends on 8)

The method of claim 8, the method further comprising training the machine learning model on the first training data set to identify a sub-set of correct quantized latent speech representation in a set of distractors by performing contrastive loss between a set of negative latent representations, a set of positive latent representations, and a set of masked contextual representations.

### Claim 13 (depends on 8)

The method of claim 8, the method further comprising:
accessing a codebook comprising a set of codewords for discrete latent representations; and
aligning each codeword included the set of codewords to a phoneme label included in a set of phoneme labels.

### Claim 14 (depends on 13)

The method of claim 13, the method further comprising:
calculating a probability that a particular codeword corresponds to a particular phoneme label; and
aligning the particular codeword to the particular phoneme label when the probability meets or exceeds a pre-defined threshold.

### Claim 15

One or more hardware storage devices that store instructions that are executable by one or more processors to cause the one or more processors to:
obtain a first training data set comprising labeled speech data or both labeled and unlabeled data corresponding to a high-resource data set, as well as latent speech representations based on the first training data set;
train a machine learning model on the first training data set to learn phonetically aware speech representations corresponding to the first training data set;
apply the latent speech representations from the machine learning model to a transformer context network to generate contextual representations;
align each contextual representation included in the contextual representations with a phoneme label to generate phonetically-aware contextual representations;
cause a refinement engine to further refine the machine learning model based on a refinement dataset, wherein the refinement engine fine-tunes the machine learning model on a limited labeled dataset corresponding to a low-resource target language and/or target domain; and
transform at least some of the contextual representations by randomly replacing a sub-set of the contextual representations with quantized latent speech representations.

### Claim 16 (depends on 15)

The one or more hardware storage devices of claim 15, the instructions being further executable to cause the one or more processors to:
before refining the machine learning model on the refinement dataset, which operates to adapt the machine learning model to the target domain, freeze or unfreeze a feature extractor of the machine learning model based on a data size of the refinement dataset.

### Claim 17 (depends on 15)

The one or more hardware storage devices of claim 15, the instructions being further executable to cause the one or more processors to:
obtain a set of contextual representations based on the first training data set;
train the machine learning model on the first training data set to align the set of contextual representations to a set of phoneme labels such that each contextual representation corresponds to a phoneme label;
obtain a set of latent representations based on the first training data set; and
train the machine learning model on the first training data set to align the set of latent representations to the set of contextual representations.

### Claim 18 (depends on 15)

The one or more hardware storage devices of claim 15, the instructions being further executable to cause the one or more processors to train the machine learning model on the first training data set to align a set of contextual representations to a set of phoneme labels such that each contextual representation corresponds to the phoneme label by performing connectionist temporal classification loss at a sequence level.

### Claim 19 (depends on 15)

The one or more hardware storage devices of claim 15, the instructions being further executable to cause the one or more processors to train the machine learning model on the first training data set to identify a sub-set of correct quantized latent speech representation in a set of distractors by performing contrastive loss between a set of negative latent representations, a set of positive latent representations, and a set of masked contextual representations.

### Claim 20 (depends on 15)

The one or more hardware storage devices of claim 15, the instructions being further executable to cause the one or more processors to:
access a codebook comprising a set of codewords for discrete latent representations; and
align each codeword included the set of codewords to a phoneme label included in a set of phoneme labels.