US 11,538,481 B2Granted

Speech segmentation based on combination of pause detection and speaker diarization

AssigneeSAS Institute Inc.

InventorsXiaolong Li, Samuel Norris Henderson, Xiaozhuo Cheng, Xu Yang

Life of this patent

Estimating the term…

Loading patent…

Abstract

An apparatus includes at least one processor to, in response to a request to perform speech-to-text conversion: perform a pause detection technique including analyzing speech audio to identify pauses, and analyzing lengths of the pauses to identify likely sentence pauses; perform a speaker diarization technique including dividing the speech audio into fragments, analyzing vocal characteristics of speech sounds of each fragment to identify a speaker of a set of speakers, and identifying instances of a change in speakers between each temporally consecutive pair of fragments to identify likely speaker changes; and perform speech-to-text operations including dividing the speech audio into segments based on at least the likely sentence pauses and likely speaker changes, using at least an acoustic model with each segment to identify likely speech sounds in the speech audio, and generating a transcript of the speech audio based at least on the likely speech sounds.

Claims27

Dependent claims

An apparatus comprising at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

receive, from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first pause detection technique comprising:
analyze the speech audio to identify pauses between speech sounds;
and
analyze lengths of the pauses to identify a first set of likely sentence pauses;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first speaker diarization technique comprising:
instantiate a speaker diarization neural network trained to output indications of vocal characteristics;
divide the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio;
for each data fragment, analyze vocal characteristics of speech sounds of the fragment by providing each data fragment of the set of data fragments to the speaker diarization neural network as an input to cause the speaker diarization neural network to generate a corresponding speaker vector of a set of speaker vectors as an output, wherein each speaker vector comprises indications of vocal characteristics detected within the corresponding data fragment;
cluster the speaker vectors of the set of speaker vectors to identify speakers;
and
identify instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes;
and
in response to the request, the at least one processor is caused to perform speech-to-text operations comprising:
based on at least the first set of likely sentence pauses, and based on at least the first set of likely speaker changes, divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
use at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
and
generate a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmit an indication of the generation of the transcript to the requesting device.

15 more elements

receive, from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first pause detection technique comprising:
derive an audio noise level based on at least one measure of a level of audio noise of the speech audio;
analyze the speech audio to identify pauses between speech sounds;
and
analyze lengths of the pauses to identify a first set of likely sentence pauses;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first speaker diarization technique comprising:
divide the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio;
for each data fragment, analyze vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers;
and
identify instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes;
and
in response to the request, the at least one processor is caused to perform preprocessing operations of a second speaker diarization technique to identify a second set of likely speaker changes;
after performing the preprocessing operations of the first speaker diarization technique and the second speaker diarization technique, the at least one processor is caused to, based on the audio noise level, and based on differences in susceptibility to inaccuracies in identifying speakers caused by audio noise between the first speaker diarization technique and the second speaker diarization technique, derive a first relative weighting between the first set of likely speaker changes and the second set of likely speaker changes;
and
in response to the request, the at least one processor is caused to perform speech-to-text operations comprising:
use the first relative weighting to combine the first set of likely speaker changes and the second set of likely speaker changes to generate a single set of indications of likely speaker changes;
based on at least the single set of indications of likely speaker changes, divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
use at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
and
generate a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmit an indication of the generation of the transcript to the requesting device.

18 more elements

receive, from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first pause detection technique comprising:
divide the speech data set into multiple data chunks that each represent a chunk of multiple chunks of the speech audio;
instantiate an instance of an acoustic model neural network comprising connectionist temporal classification (CTC) output;
analyze the speech audio to identify pauses between speech sounds by providing each data chunk of the multiple data chunks to the instance of the acoustic model neural network as an input and monitor the CTC output for at least one corresponding string of blank symbols indicative of a pause;
and
analyze lengths of the pauses to identify a first set of likely sentence pauses by comparing a length of each string of blank symbols from the CTC output to a predetermined blank threshold length;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first speaker diarization technique comprising:
divide the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio;
for each data fragment, analyze vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers;
and
identify instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes;
and
in response to the request, the at least one processor is caused to perform speech-to-text operations comprising:
based on at least the first set of likely sentence pauses, and based on at least the first set of likely speaker changes, divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
use at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
and
generate a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmit an indication of the generation of the transcript to the requesting device.

15 more elements

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:

receive, from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first pause detection technique comprising:
analyze the speech audio to identify pauses between speech sounds;
and
analyze lengths of the pauses to identify a first set of likely sentence pauses;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first speaker diarization technique comprising:
instantiate a speaker diarization neural network trained to output indications of vocal characteristics;
divide the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio;
for each data fragment, analyze vocal characteristics of speech sounds of the fragment by providing each data fragment of the set of data fragments to the speaker diarization neural network as an input to cause the speaker diarization neural network to generate a corresponding speaker vector of a set of speaker vectors as an output, wherein each speaker vector comprises indications of vocal characteristics detected within the corresponding data fragment;
cluster the speaker vectors of the set of speaker vectors to identify speakers;
and
identify instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes;
and
in response to the request, the at least one processor is caused to perform speech-to-text operations comprising:
based on at least the first set of likely sentence pauses, and based on at least the first set of likely speaker changes, divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
use at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
and
generate a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmit an indication of the generation of the transcript to the requesting device.

15 more elements

receive, from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first pause detection technique comprising:
derive an audio noise level based on at least one measure of a level of audio noise of the speech audio;
analyze the speech audio to identify pauses between speech sounds;
and
analyze lengths of the pauses to identify a first set of likely sentence pauses;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first speaker diarization technique comprising:
divide the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio;
for each data fragment, analyze vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers;
and
identify instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes;
and
in response to the request, the at least one processor is caused to perform preprocessing operations of a second speaker diarization technique to identify a second set of likely speaker changes;
after performing the preprocessing operations of the first speaker diarization technique and the second speaker diarization technique, the at least one processor is caused to, based on the audio noise level, and based on differences in susceptibility to inaccuracies in identifying speakers caused by audio noise between the first speaker diarization technique and the second speaker diarization technique, derive a first relative weighting between the first set of likely speaker changes and the second set of likely speaker changes;
and
in response to the request, the at least one processor is caused to perform speech-to-text operations comprising:
use the first relative weighting to combine the first set of likely speaker changes and the second set of likely speaker changes to generate a single set of indications of likely speaker changes;
based on at least the single set of indications of likely speaker changes, divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
use at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
and
generate a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmit an indication of the generation of the transcript to the requesting device.

18 more elements

receive, from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first pause detection technique comprising:
divide the speech data set into multiple data chunks that each represent a chunk of multiple chunks of the speech audio;
instantiate an instance of an acoustic model neural network comprising connectionist temporal classification (CTC) output;
analyze the speech audio to identify pauses between speech sounds by providing each data chunk of the multiple data chunks to the instance of the acoustic model neural network as an input and monitor the CTC output for at least one corresponding string of blank symbols indicative of a pause;
and
analyze lengths of the pauses to identify a first set of likely sentence pauses by comparing a length of each string of blank symbols from the CTC output to a predetermined blank threshold length;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first speaker diarization technique comprising:
divide the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio;
for each data fragment, analyze vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers;
and
identify instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes;
and
in response to the request, the at least one processor is caused to perform speech-to-text operations comprising:
based on at least the first set of likely sentence pauses, and based on at least the first set of likely speaker changes, divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
use at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
and
generate a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmit an indication of the generation of the transcript to the requesting device.

15 more elements

A computer-implemented method comprising:

receiving, by at least one processor, and from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, performing preprocessing operations of a first pause detection technique comprising:
analyzing, by the at least one processor, the speech audio to identify pauses between speech sounds;
and
analyzing, by the at least one processor, lengths of the pauses to identify a first set of likely sentence pauses;
in response to the request, performing preprocessing operations of a first speaker diarization technique comprising:
instantiating, by the at least one processor, a speaker diarization neural network trained to output indications of vocal characteristics;
dividing the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio;
for each data fragment, analyzing, by the at least one processor, vocal characteristics of speech sounds of the fragment by providing each data fragment of the set of data fragments to the speaker diarization neural network as an input to cause the speaker diarization neural network to generate a corresponding speaker vector of a set of speaker vectors as an output, wherein each speaker vector comprises indications of vocal characteristics detected within the corresponding data fragment;
clustering, by the at least one processor, the speaker vectors of the set of speaker vectors to identify the speakers of the set of speakers;
identifying, by the at least one processor, instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes;
and
in response to the request, performing speech-to-text operations comprising:
based on at least the first set of likely sentence pauses, and based on at least the first set of likely speaker changes, dividing the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
using, by the at least one processor, at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
and
generating, by the at least one processor, a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmitting, from the at least one processor, an indication of the generation of the transcript to the requesting device.

14 more elements

A computer-implemented method comprising:

receiving, by at least one processor, and from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, performing preprocessing operations of a first pause detection technique comprising:
deriving, by the at least one processor, an audio noise level based on at least one measure of a level of audio noise of the speech audio;
analyzing, by the at least one processor, the speech audio to identify pauses between speech sounds;
and
analyzing, by the at least one processor, lengths of the pauses to identify a first set of likely sentence pauses;
in response to the request, performing preprocessing operations of a first speaker diarization technique comprising:
dividing the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio;
for each data fragment, analyzing, by the at least one processor, vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers;
and
identifying, by the at least one processor, instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes;
and
in response to the request, performing, by the at least one processor, preprocessing operations of a second speaker diarization technique to identify a second set of likely speaker changes;
after performing the preprocessing operations of the first speaker diarization technique and the second speaker diarization technique, deriving, by the at least one processor, a first relative weighting between the first set of likely speaker changes and the second set of likely speaker changes based on the audio noise level, and based on differences in susceptibility to inaccuracies in identifying speakers caused by audio noise between the first speaker diarization technique and the second speaker diarization technique;
and
in response to the request, performing speech-to-text operations comprising:
using, by the at least one processor, the first relative weighting to combine the first set of likely speaker changes and the second set of likely speaker changes to generate a single set of indications of likely speaker changes;
based on at least the single set of indications of likely speaker changes, dividing the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
using, by the at least one processor, at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
and
generating, by the at least one processor, a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmitting, from the at least one processor, an indication of the generation of the transcript to the requesting device.

18 more elements

A computer-implemented method comprising:

receiving, by at least one processor, and from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, performing preprocessing operations of a first pause detection technique comprising:
dividing the speech data set into multiple data chunks that each represent a chunk of multiple chunks of the speech audio;
instantiating, by the at least one processor, an instance of an acoustic model neural network comprising connectionist temporal classification (CTC) output;
analyzing, by the at least one processor, the speech audio to identify pauses between speech sounds by providing each data chunk of the multiple data chunks to the instance of the acoustic model neural network as an input and monitoring the CTC output for at least one corresponding string of blank symbols indicative of a pause;
and
analyzing, by the at least one processor, lengths of the pauses to identify a first set of likely sentence pauses by comparing a length of each string of blank symbols from the CTC output to a predetermined blank threshold length;
in response to the request, performing preprocessing operations of a first speaker diarization technique comprising:
dividing the speech data set into a set of data fragments that each represent a fragment of a set of fragments of the speech audio;
for each data fragment, analyzing, by the at least one processor, vocal characteristics of speech sounds of the fragment to identify a speaker of a set of speakers;
and
identifying, by the at least one processor, instances of a change in speakers between temporally consecutive fragments of the set of fragments to identify a first set of likely speaker changes;
and
in response to the request, performing speech-to-text operations comprising:
based on at least the first set of likely sentence pauses, and based on at least the first set of likely speaker changes, dividing the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
using, by the at least one processor, at least an acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
and
generating, by the at least one processor, a transcript of the speech data set based, at least in part, on the identified likely speech sounds, or transmitting, from the at least one processor, an indication of the generation of the transcript to the requesting device.

15 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/851,264Filed 28 Jun 2022 · continuation in part
Priority: 18 Mar 2020
Published: US 2022/0028395 A120 Oct 2022
Granted: 27 Dec 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification13 codes

G06N 3/045
Architecture, e.g. interconnection topology: combinations of networks
G10L 15/26
Speech recognition: speech to text systems
G06N 3/0499
Architecture, e.g. interconnection topology: feedforward networks
G10L 15/02
Speech recognition: feature extraction for speech recognition; Selection of recognition unit

Family

Application62/991,275Us provisional application · filed 18 Mar 2020
Application17/138,521Parent of this continuation-in-part · filed 30 Dec 2020
Application17/138,445Parent of this continuation · filed 30 Dec 2020
Application17/205,871Parent of this continuation-in-part · filed 18 Mar 2021
PCT/CN2021/082572Parent of this continuation · filed 24 Mar 2021
US 11,049,502 B1Parent of this continuation-in-part · filed 29 Jun 2021
Application17/370,441Parent of this continuation-in-part · filed 8 Jul 2021
US 11,138,979 B1Parent of this continuation · filed 5 Oct 2021
Application17/498,811Parent of this continuation-in-part · filed 12 Oct 2021
US 11,145,309 B1Parent of this continuation-in-part · filed 12 Oct 2021
Application63/288,385Us provisional application · filed 10 Dec 2021
Application63/297,002Us provisional application · filed 6 Jan 2022
US 2022/0335947 A1Pre-grant publication · published 20 Oct 2022
US 11,373,655 B2Parent of this continuation-in-part
US 11,404,053 B1Parent of this continuation-in-part
Application17/370,441Parent of this continuation
Application17/138,521Parent of this continuation
US 11,217,233 B1Related patent
US 2021/0295845 A1Published application
US 2021/0295846 A1Published application
US 2022/0028395 A1Published application
US 11,538,481 B2This patent

From the US filing history. Foreign family members aren’t listed.