# US 2026/0065901 A1 · Speech pre-training methods, apparatuses, storage media, and electronic devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0065901 A1
- Document ID: US20260065901A1
- Status: Published application
- Kind: A1
- Application number: 19/287,922
- Filed: 2025-08-01
- Published: 2026-03-05
- Earliest claimed: 2024-08-27
- Assignees: Alipay (Hangzhou) Information Technology Co., Ltd.
- Applicants: Alipay (Hangzhou) Information Technology Co., Ltd.
- Inventors: Anqi LU; Zhiming WANG
- Web page: https://patentgrep.com/patent/US20260065901A1

## CPC

- G10L 15/063: {Training}
- G10L 25/24: the extracted parameters being the cepstrum
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/16: using artificial neural networks
- G10L 17/04: Training, enrolment or model building
- G10L 25/30: using neural networks
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L 2015/0631: {Creating reference templates; Clustering}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Described is speech pre-training, which includes acquiring a speech sample and phoneme data corresponding to the speech sample. Speech features of speech frames are extracted in the speech sample. Based on the speech features and the phoneme data, the speech sample is divided into at least one speech segment, where one speech segment corresponds to one phoneme. Based on speech features of speech frames in speech segments corresponding to a same phoneme, target features of phonemes are determined. The target features of the phonemes are used as initial clustering centers. Based on the initial clustering centers and to obtain corresponding clustering labels, the speech features of the speech frames in the speech sample are clustered. By using the corresponding clustering labels to obtain a speech pre-training model, a predetermined network model is trained.

## Claims

### Claim 1

A computer-implemented method for speech pre-training, comprising:
acquiring a speech sample and phoneme data corresponding to the speech sample;
extracting speech features of speech frames in the speech sample;
dividing, based on the speech features and the phoneme data, the speech sample into at least one speech segment, wherein one speech segment corresponds to one phoneme;
determining, based on speech features of speech frames in speech segments corresponding to a same phoneme, target features of phonemes;
using, as initial clustering centers, the target features of the phonemes;
clustering, based on the initial clustering centers and to obtain corresponding clustering labels, the speech features of the speech frames in the speech sample; and
training, by using the corresponding clustering labels to obtain a speech pre-training model, a predetermined network model.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein dividing, based on the speech features and the phoneme data, the speech sample into at least one speech segment, comprises:
aligning, to determine a speech segment corresponding to each phoneme in a time dimension, the speech features of the speech frames in the speech sample with the phoneme data.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein determining, based on speech features of speech frames in speech segments corresponding to a same phoneme, target features of phonemes, comprises:
averaging, to obtain the target features of the phonemes, the speech features of the speech frames in the speech segments corresponding to the same phoneme.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein clustering, based on the initial clustering centers and to obtain corresponding clustering labels, the speech features of the speech frames in the speech sample, comprises:
calculating distances between the initial clustering centers and the speech features of the speech frames in the speech sample;
determining, based on the distances and to allocate the speech frames in the speech sample to a speech feature cluster that an initial clustering center is located in, the initial clustering center closest to the speech frames in the speech sample;
updating cluster centers of speech feature clusters; and
clustering, based on updated cluster centers, until a clustering termination condition is satisfied, and to obtain the corresponding clustering labels corresponding to the speech frames in the speech sample, the speech frames in the speech sample.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein updating cluster centers of speech feature clusters, comprises:
averaging, to obtain a corresponding average feature, speech features of all speech frames in each speech feature cluster; and
using the corresponding average feature as an updated cluster center of each speech feature cluster.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein:
training, by using the corresponding clustering labels to obtain a speech pre-training model, a predetermined network model comprises:
randomly masking, to obtain corresponding masked features, the speech features of the speech frames in the speech sample;
inputting, to obtain prediction labels of the speech frames, the corresponding masked features to the predetermined network model;
calculating loss values between the prediction labels and the corresponding clustering labels; and
iteratively training, to obtain the speech pre-training model, based on the loss values, and until an iteration termination condition is satisfied, model parameters of the predetermined network model.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein:
the predetermined network model comprises an encoder and a prediction head; and
inputting, to obtain prediction labels of the speech frames, the corresponding masked features to the predetermined network model, comprises:
inputting, to obtain corresponding hidden states, the corresponding masked features to the encoder; and
mapping, to obtain the prediction labels of the speech frames and by using the prediction head, the corresponding hidden states to label space.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein:
the speech features comprise first Mel spectrum features; and
extracting speech features of speech frames in the speech sample, comprises:
performing frame division processing on the speech sample; and
extracting the first Mel spectrum features of the speech frames.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the speech features comprise second Mel spectrum features; and
extracting speech features of speech frames in the speech sample, comprises:
inputting, to obtain the second Mel spectrum features of the speech frames in the speech sample, a text sequence corresponding to the speech sample to a trained feature extraction model.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein:
the trained feature extraction model comprises at least a first encoder, a second encoder, and a decoder; and
inputting, to obtain the second Mel spectrum features of the speech frames in the speech sample, a text sequence corresponding to the speech sample to a trained feature extraction model, comprises:
performing, by using the first encoder and to obtain first latent variable features feature, extraction on the text sequence;
performing, by using the second encoder and to obtain second latent variable features, feature extraction on a linear spectrum of the speech sample; and
performing, by using the decoder and to obtain the second Mel spectrum features of the speech frames in the speech sample, decoding processing on the second latent variable features.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein acquiring a speech sample and phoneme data corresponding to the speech sample, comprises:
acquiring the speech sample and a natural text corresponding to the speech sample; and
performing, to obtain the phoneme data, phoneme processing on the natural text.

### Claim 12

A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations for speech pre-training, comprising:
acquiring a speech sample and phoneme data corresponding to the speech sample;
extracting speech features of speech frames in the speech sample;
dividing, based on the speech features and the phoneme data, the speech sample into at least one speech segment, wherein one speech segment corresponds to one phoneme;
determining, based on speech features of speech frames in speech segments corresponding to a same phoneme, target features of phonemes;
using, as initial clustering centers, the target features of the phonemes;
clustering, based on the initial clustering centers and to obtain corresponding clustering labels, the speech features of the speech frames in the speech sample; and
training, by using the corresponding clustering labels to obtain a speech pre-training model, a predetermined network model.

### Claim 13 (depends on 12)

The non-transitory, computer-readable medium of claim 12, wherein dividing, based on the speech features and the phoneme data, the speech sample into at least one speech segment, comprises:
aligning, to determine a speech segment corresponding to each phoneme in a time dimension, the speech features of the speech frames in the speech sample with the phoneme data.

### Claim 14 (depends on 12)

The non-transitory, computer-readable medium of claim 12, wherein determining, based on speech features of speech frames in speech segments corresponding to a same phoneme, target features of phonemes, comprises:
averaging, to obtain the target features of the phonemes, the speech features of the speech frames in the speech segments corresponding to the same phoneme.

### Claim 15 (depends on 12)

The non-transitory, computer-readable medium of claim 12, wherein clustering, based on the initial clustering centers and to obtain corresponding clustering labels, the speech features of the speech frames in the speech sample, comprises:
calculating distances between the initial clustering centers and the speech features of the speech frames in the speech sample;
determining, based on the distances and to allocate the speech frames in the speech sample to a speech feature cluster that an initial clustering center is located in, the initial clustering center closest to the speech frames in the speech sample;
updating cluster centers of speech feature clusters; and
clustering, based on updated cluster centers, until a clustering termination condition is satisfied, and to obtain the corresponding clustering labels corresponding to the speech frames in the speech sample, the speech frames in the speech sample.

### Claim 16 (depends on 15)

The non-transitory, computer-readable medium of claim 15, wherein updating cluster centers of speech feature clusters, comprises:
averaging, to obtain a corresponding average feature, speech features of all speech frames in each speech feature cluster; and
using the corresponding average feature as an updated cluster center of each speech feature cluster.

### Claim 17 (depends on 12)

The non-transitory, computer-readable medium of claim 12, wherein:
training, by using the corresponding clustering labels to obtain a speech pre-training model, a predetermined network model comprises:
randomly masking, to obtain corresponding masked features, the speech features of the speech frames in the speech sample;
inputting, to obtain prediction labels of the speech frames, the corresponding masked features to the predetermined network model;
calculating loss values between the prediction labels and the corresponding clustering labels; and
iteratively training, to obtain the speech pre-training model, based on the loss values, and until an iteration termination condition is satisfied, model parameters of the predetermined network model.

### Claim 18 (depends on 17)

The non-transitory, computer-readable medium of claim 17, wherein:
the predetermined network model comprises an encoder and a prediction head; and
inputting, to obtain prediction labels of the speech frames, the corresponding masked features to the predetermined network model, comprises:
inputting, to obtain corresponding hidden states, the corresponding masked features to the encoder; and
mapping, to obtain the prediction labels of the speech frames and by using the prediction head, the corresponding hidden states to label space.

### Claim 19 (depends on 12)

The non-transitory, computer-readable medium of claim 12, wherein:
the speech features comprise first Mel spectrum features; and
extracting speech features of speech frames in the speech sample, comprises:
performing frame division processing on the speech sample; and
extracting the first Mel spectrum features of the speech frames.

### Claim 20

A computer-implemented system for speech pre-training, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations, comprising:
acquiring a speech sample and phoneme data corresponding to the speech sample;
extracting speech features of speech frames in the speech sample;
dividing, based on the speech features and the phoneme data, the speech sample into at least one speech segment, wherein one speech segment corresponds to one phoneme;
determining, based on speech features of speech frames in speech segments corresponding to a same phoneme, target features of phonemes;
using, as initial clustering centers, the target features of the phonemes;
clustering, based on the initial clustering centers and to obtain corresponding clustering labels, the speech features of the speech frames in the speech sample; and
training, by using the corresponding clustering labels to obtain a speech pre-training model, a predetermined network model.
