# US 12,183,362 B1 · Speech recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,362 B1
- Document ID: US12183362B1
- Status: Granted
- Kind: B1
- Application number: 18/633,370
- Filed: 2024-04-11
- Granted: 2024-12-31
- Earliest claimed: 2023-07-20
- Assignees: Mashang Consumer Finance Co., Ltd.
- Applicants: Mashang Consumer Finance Co., Ltd.
- Inventors: Qinglin Meng; Bin Yang; Ning Jiang; Haiying Wu; Quan Lu; Min Liu
- Web page: https://patentgrep.com/patent/US12183362B1

## CPC

- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 17/00: Speaker identification or verification techniques
- G10L 17/04: Training, enrolment or model building
- G10L 17/18: Artificial neural networks; Connectionist approaches
- G10L 15/08: Speech classification or search
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A speech recognition method. The method includes: performing speech activity detection on speech data to obtain multiple speech segments; determining, for each of the speech segments, a number of speakers involved in the each of the speech segments; for each of at least one of the speech segments with the determined number greater than 1: performing speech separation on the each of at least one of the speech segments to obtain multiple audio segments; performing speech recognition on each of the audio segments to obtain respective first speech recognition results for the audio segments; performing feature extraction on each of the audio segments to obtain respective voiceprint feature vectors; and performing clustering on the audio segments with respect to the speakers to obtain a clustering result; and obtaining a second speech recognition result for the speech data based on the clustering result and the respective first speech recognition results.

## Claims

### Claim 1

A speech recognition method, comprising: by an electronic device,
performing speech activity detection on speech data to obtain a plurality of speech segments;
determining, for each of the plurality of speech segments, a number of speakers involved in the each of the plurality of speech segments;
for each of at least one of the plurality of speech segments with the determined number of speakers greater than 1:
performing speech separation on the each of at least one of the plurality of speech segments to obtain a plurality of audio segments;
performing speech recognition on each of the plurality of audio segments to obtain respective first speech recognition results for the plurality of audio segments;
performing feature extraction on each of the plurality of audio segments to obtain respective voiceprint feature vectors of the plurality of audio segments; and
performing clustering on the plurality of audio segments with respect to the speakers based on the respective voiceprint feature vectors to obtain a clustering result indicating that each of the speakers corresponds to which of the plurality of audio segments; and
obtaining a second speech recognition result for the speech data based on the clustering result and the respective first speech recognition results for the each of at least one of the plurality of speech segments.

### Claim 2 (depends on 1)

The speech recognition method according to claim 1, further comprising: by the electronic device,
for any one of the plurality of speech segments with the determined number of speakers equal to 1, taking the any one of the plurality of speech segments as an audio segment corresponding to one speaker.

### Claim 3 (depends on 1)

The speech recognition method according to claim 1, wherein the determining of the number of speakers involved in the each of the plurality of speech segments comprises:
using a speaker division model to determine the number of speakers involved in the each of the plurality of speech segments by:
extracting a voiceprint feature and a Mel-scale Frequency Cepstral Coefficients feature from the each of the plurality of speech segments;
encoding the Mel-scale Frequency Cepstral Coefficients feature to obtain an encoded feature vector;
decoding the encoded feature vector to obtain a decoded feature vector;
merging the decoded feature vector and a vector of the voiceprint feature to obtain a prediction feature vector; and
predicting a number of speakers based on the prediction feature vector to obtain the number of speakers involved in the each of the plurality of speech segments.

### Claim 4 (depends on 3)

The speech recognition method according to claim 3, wherein the extracting of the voiceprint feature and the Mel-scale Frequency Cepstral Coefficients feature comprises:
extracting the voiceprint feature from the each of the plurality of speech segments by a voiceprint feature network; and
extracting the Mel-scale Frequency Cepstral Coefficients feature by a Mel-scale Frequency Cepstral Coefficients network.

### Claim 5 (depends on 1)

The speech recognition method according to claim 1, wherein the performing of the speech recognition on each of the plurality of audio segments comprises:
performing the speech recognition on each of the plurality of audio segments by a speech recognition model; and
the speech recognition model is obtained by:
performing augmentation processing on a speech recognition sample to obtain an augmented speech recognition sample, and obtaining a speech recognition training sample comprising the speech recognition sample and the augmented speech recognition sample;
inputting the speech recognition training sample into a first speech recognition model to obtain a sample speech recognition result for the speech recognition training sample;
determining a speech recognition loss value based on the sample speech recognition result and a preset loss function; and
in response to determining that the speech recognition loss value is less than or equal to a preset loss threshold, taking the first speech recognition model as the speech recognition model.

### Claim 6 (depends on 5)

The speech recognition method according to claim 5, wherein the performing of the speech separation on the each of at least one of the plurality of speech segments to obtain the plurality of audio segments comprises:
using a speech separation model to perform the speech separation on the each of at least one of the plurality of speech segments to obtain the plurality of audio segments by:
performing short-time Fourier transform on the each of at least one of the plurality of speech segments to obtain a complex spectrum;
performing power-law compression on the complex spectrum to obtain a spectrogram;
performing network coding on the spectrogram to obtain a network coding result;
performing mapping coding on the network coding result to obtain a mapping coding result;
performing first processing on the mapping coding result to obtain predicted phase information and predicted amplitude information of the each of at least one of the plurality of speech segments;
performing second processing on the mapping coding result to obtain a real part and an imaginary part of the each of at least one of the plurality of speech segments;
obtaining an enhanced complex spectrum with an enhanced amplitude based on the predicted phase information, the predicted amplitude information, the real part and the imaginary part; and
performing inverse short-time Fourier transform on the enhanced complex spectrum to obtain the plurality of audio segments.

### Claim 7 (depends on 6)

The speech recognition method according to claim 6, wherein the speech recognition training sample comprises a plurality of audio segments output by the speech separation model.

### Claim 8 (depends on 1)

The speech recognition method according to claim 1, wherein the performing of the clustering on the plurality of audio segments with respect to the speakers based on the respective voiceprint feature vectors to obtain the clustering result comprises:
normalizing each of the respective voiceprint feature vectors to obtain respective normalized feature vectors of the plurality of audio segments;
transposing each of the respective normalized feature vectors to obtain respective transposed feature vectors of the plurality of audio segments;
calculating an inner product of each of the respective normalized feature vectors and one of the respective transposed feature vectors corresponding to the each of the respective normalized feature vectors, to obtain a cross similarity matrix of the plurality of audio segments, wherein the cross similarity matrix comprises a plurality of rows each representing a class; and
performing clustering operation based on the cross similarity matrix to obtain a new cross similarity matrix until the new cross similarity matrix meets a clustering termination condition to obtain the clustering result; and
the performing of the clustering operation comprises:
obtain one or more distances for the cross similarity matrix by: for every two rows of the plurality of rows in the cross similarity matrix, calculating one of the one or more distances between the every two rows;
determining two ones of the plurality of rows having one of the one or more distances having a smallest value; and
merging the determined two ones of the plurality of rows to obtain the new cross similarity matrix.

### Claim 9 (depends on 8)

The speech recognition method according to claim 8, wherein the clustering termination condition comprises only two rows remaining in the new cross similarity matrix.

### Claim 10 (depends on 1)

The speech recognition method according to claim 1, wherein the obtaining of the second speech recognition result for the speech data comprises:
associating, for each of one or more speakers, one or more audio segments in the speech data corresponding to the each of one or more speakers with the each of one or more speakers based on the clustering result for the each of at least one of the plurality of speech segments, to obtain the second speech recognition result.

### Claim 11

An electronic device, comprising:
a processor; and
a memory storing a computer program executable by the processor to perform operations comprising:
performing speech activity detection on speech data to obtain a plurality of speech segments;
determining, for each of the plurality of speech segments, a number of speakers involved in the each of the plurality of speech segments;
for each of at least one of the plurality of speech segments with the determined number of speakers greater than 1:
performing speech separation on the each of at least one of the plurality of speech segments to obtain a plurality of audio segments;
performing speech recognition on each of the plurality of audio segments to obtain respective first speech recognition results for the plurality of audio segments;
performing feature extraction on each of the plurality of audio segments to obtain respective voiceprint feature vectors of the plurality of audio segments; and
performing clustering on the plurality of audio segments with respect to the speakers based on the respective voiceprint feature vectors to obtain a clustering result indicating that each of the speakers corresponds to which of the plurality of audio segments; and
obtaining a second speech recognition result for the speech data based on the clustering result and the respective first speech recognition results for the each of at least one of the plurality of speech segments.

### Claim 12 (depends on 11)

The electronic device according to claim 11, wherein the operations further comprise:
for any one of the plurality of speech segments with the determined number of speakers equal to 1, taking the any one of the plurality of speech segments as an audio segment corresponding to one speaker.

### Claim 13 (depends on 11)

The electronic device according to claim 11, wherein the determining of the number of speakers involved in the each of the plurality of speech segments comprises:
using a speaker division model to determine the number of speakers involved in the each of the plurality of speech segments by:
extracting a voiceprint feature and a Mel-scale Frequency Cepstral Coefficients feature from the each of the plurality of speech segments;
encoding the Mel-scale Frequency Cepstral Coefficients feature to obtain an encoded feature vector;
decoding the encoded feature vector to obtain a decoded feature vector;
merging the decoded feature vector and a vector of the voiceprint feature to obtain a prediction feature vector; and
predicting a number of speakers based on the prediction feature vector to obtain the number of speakers involved in the each of the plurality of speech segments.

### Claim 14 (depends on 13)

The electronic device according to claim 13, wherein the extracting of the voiceprint feature and the Mel-scale Frequency Cepstral Coefficients feature comprises:
extracting the voiceprint feature from the each of the plurality of speech segments by a voiceprint feature network; and
extracting the Mel-scale Frequency Cepstral Coefficients feature by a Mel-scale Frequency Cepstral Coefficients network.

### Claim 15 (depends on 11)

The electronic device according to claim 11, wherein the performing of the speech recognition on each of the plurality of audio segments comprises:
performing the speech recognition on each of the plurality of audio segments by a speech recognition model; and
the speech recognition model is obtained by:
performing augmentation processing on a speech recognition sample to obtain an augmented speech recognition sample, and obtaining a speech recognition training sample comprising the speech recognition sample and the augmented speech recognition sample;
inputting the speech recognition training sample into a first speech recognition model to obtain a sample speech recognition result for the speech recognition training sample;
determining a speech recognition loss value based on the sample speech recognition result and a preset loss function; and
in response to determining that the speech recognition loss value is less than or equal to a preset loss threshold, taking the first speech recognition model as the speech recognition model.

### Claim 16 (depends on 15)

The electronic device according to claim 15, wherein the performing of the speech separation on the each of at least one of the plurality of speech segments to obtain the plurality of audio segments comprises:
using a speech separation model to perform the speech separation on the each of at least one of the plurality of speech segments to obtain the plurality of audio segments by:
performing short-time Fourier transform on the each of at least one of the plurality of speech segments to obtain a complex spectrum;
performing power-law compression on the complex spectrum to obtain a spectrogram;
performing network coding on the spectrogram to obtain a network coding result;
performing mapping coding on the network coding result to obtain a mapping coding result;
performing first processing on the mapping coding result to obtain predicted phase information and predicted amplitude information of the each of at least one of the plurality of speech segments;
performing second processing on the mapping coding result to obtain a real part and an imaginary part of the each of at least one of the plurality of speech segments;
obtaining an enhanced complex spectrum with an enhanced amplitude based on the predicted phase information, the predicted amplitude information, the real part and the imaginary part; and
performing inverse short-time Fourier transform on the enhanced complex spectrum to obtain the plurality of audio segments.

### Claim 17 (depends on 16)

The electronic device according to claim 16, wherein the speech recognition training sample comprises a plurality of audio segments output by the speech separation model.

### Claim 18 (depends on 11)

The electronic device according to claim 11, wherein the performing of the clustering on the plurality of audio segments with respect to the speakers based on the respective voiceprint feature vectors to obtain the clustering result comprises:
normalizing each of the respective voiceprint feature vectors to obtain respective normalized feature vectors of the plurality of audio segments;
transposing each of the respective normalized feature vectors to obtain respective transposed feature vectors of the plurality of audio segments;
calculating an inner product of each of the respective normalized feature vectors and one of the respective transposed feature vectors corresponding to the each of the respective normalized feature vectors, to obtain a cross similarity matrix of the plurality of audio segments, wherein the cross similarity matrix comprises a plurality of rows each representing a class; and
performing clustering operation based on the cross similarity matrix to obtain a new cross similarity matrix until the new cross similarity matrix meets a clustering termination condition to obtain the clustering result; and
the performing of the clustering operation comprises:
obtain one or more distances for the cross similarity matrix by: for every two rows of the plurality of rows in the cross similarity matrix, calculating one of the one or more distances between the every two rows;
determining two ones of the plurality of rows having one of the one or more distances having a smallest value; and
merging the determined two ones of the plurality of rows to obtain the new cross similarity matrix.

### Claim 19 (depends on 18)

The electronic device according to claim 18, wherein the clustering termination condition comprises only two rows remaining in the new cross similarity matrix.

### Claim 20

A non-transitory computer-readable storage medium storing a computer program executable by a processor to perform operations comprising:
performing speech activity detection on speech data to obtain a plurality of speech segments;
determining, for each of the plurality of speech segments, a number of speakers involved in the each of the plurality of speech segments;
for each of at least one of the plurality of speech segments with the determined number of speakers greater than 1:
performing speech separation on the each of at least one of the plurality of speech segments to obtain a plurality of audio segments;
performing speech recognition on each of the plurality of audio segments to obtain respective first speech recognition results for the plurality of audio segments;
performing feature extraction on each of the plurality of audio segments to obtain respective voiceprint feature vectors of the plurality of audio segments; and
performing clustering on the plurality of audio segments with respect to the speakers based on the respective voiceprint feature vectors to obtain a clustering result indicating that each of the speakers corresponds to which of the plurality of audio segments; and
obtaining a second speech recognition result for the speech data based on the clustering result and the respective first speech recognition results for the each of at least one of the plurality of speech segments.