# US 2022/0028395 A1 · Dual use of acoustic model in speech-to-text framework

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0028395 A1
- Document ID: US20220028395A1
- Status: Published application
- Kind: A1
- Application number: 17/498,811
- Filed: 2021-10-12
- Published: 2022-01-27
- Earliest claimed: 2020-03-18
- Assignees: SAS Institute Inc.
- Applicants: SAS Institute Inc.
- Inventors: XIAOLONG LI; XIAOZHUO CHENG; XU YANG
- Web page: https://patentgrep.com/patent/US20220028395A1

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/048: Activation functions
- G06N 3/0499: Feedforward networks
- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/16: using artificial neural networks
- G10L 25/30: using neural networks
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G06N 3/043: based on fuzzy logic, fuzzy membership or fuzzy inference, e.g. adaptive neuro-fuzzy inference systems [ANFIS]
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- G10L 2025/783: {based on threshold decision}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An apparatus includes processor(s) to: perform preprocessing operations of a segmentation technique including divide speech data set into data chunks representing chunks of speech audio, use an acoustic model with each data chunk to identify pauses in the speech audio, and analyze a length of time of each identified pause to identify a candidate set of likely sentence pauses in the speech audio; and perform speech-to-text operations including divide the speech data set into data segments that each representing segments of the speech audio based on the candidate set of likely sentence pauses, use the acoustic model with each data segment to identify likely speech sounds in the speech audio, analyze the identified likely speech sounds to identify candidate sets of words likely spoken in the speech audio, and generate a transcript of the speech data set based at least on the candidate sets of words likely spoken.

## Claims

### Claim 1

An apparatus comprising at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receive, from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, the at least one processor is caused to perform preprocessing operations of a first segmentation technique comprising:
divide the speech data set into multiple data chunks that each represent a chunk of multiple chunks of the speech audio;
use an acoustic model with each data chunk of the multiple data chunks to identify pauses in the speech audio; and
analyze a length of time of each identified pause in the speech audio to identify a first candidate set of likely sentence pauses in the speech audio;
in response to the request, the at least one processor is caused to perform speech-to-text operations comprising:
based, at least in part, on the first candidate set of likely sentence pauses, divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
use the acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
analyze the identified likely speech sounds to identify candidate sets of words likely spoken in the speech audio;
based, at least in part, on the candidate sets of words likely spoken in the speech audio, generate a transcript of the speech data set; and
transmit an indication of the generation of the transcript to the requesting device.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein:
the acoustic model is based on a neural network;
the neural network comprises a connectionist temporal classification (CTC) output;
the CTC output generates strings of one or more blank symbols for the identified pauses;
a quantity of blank symbols in each string of one or more blank symbols correlates to a length of a corresponding identified pause; and
the identification of the first candidate set of likely sentence pauses comprises:
comparing the length of each identified speech pause indicated by a corresponding string of one or more blank symbols to a predetermined blank threshold length; and
designating each identified speech pause that is at least as long as the predetermined blank threshold length as a likely sentence pause of the first candidate set of likely sentence pauses.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein:
the acoustic model outputs indications of occurrences of speech sounds as indications of occurrences of graphemes of a set of graphemes;
the blank symbol comprises a grapheme of the set of graphemes that is indicative of a pause between speech sounds; and
the identification of candidate sets of words likely spoken in the speech audio comprises analyzing the indications of occurrence of graphemes output by the acoustic model to identify combinations of graphemes that correspond to words.

### Claim 4 (depends on 2)

The apparatus of claim 2, comprising at least one neuromorphic device to enable instantiation of the acoustic model, the at least one processor is caused to perform operations comprising:
prior to the use of the acoustic model to identify pauses in the speech audio, configure the at least one neuromorphic device to instantiate at least one instance of the acoustic model; and
following the use of the acoustic model to identify pauses in the speech audio, maintain the configuration of the at least one neuromorphic device to maintain the at least one instance of the acoustic model to enable use of the at least one instance of the acoustic model to identify speech sounds in the speech audio.

### Claim 5 (depends on 2)

The apparatus of claim 2, comprising a distributed processing system that comprises a control device and multiple node devices, wherein:
the at least one processor comprises a processor of the control device and multiple processors of the multiple node devices; and
the processor of the control device is caused to distribute indications of a configuration of the neural network of the acoustic model to the multiple node devices to enable instantiation of multiple instances of the acoustic model within the multiple node devices by the multiple processors of the multiple node devices.

### Claim 6 (depends on 2)

The apparatus of claim 2, wherein:
in response to the request, the at least one processor is caused to perform pre-processing operations of a second segmentation technique comprising:
divide the speech audio into multiple alternate data chunks that each represent an alternate chunk of multiple alternate chunks of the speech audio;
derive a threshold amplitude based on at least one peak amplitude of the speech audio;
compare a peak amplitude of the alternate chunk of the speech audio of each alternate data chunk to the threshold amplitude;
designate each alternate data chunk with a peak amplitude that is below the threshold amplitude as an alternate pause data chunk;
within at least one set of temporally consecutive alternate data chunks of the multiple alternate chunks, identify a longest subset of temporally consecutive alternate pause data chunks; and
for each set of the at least one set of temporally consecutive alternate data chunks, store an indication of the longest subset of temporally consecutive alternate pause data chunks as a likely sentence pause of a second candidate set of likely sentence pauses; and
the division of the of the speech data set into the multiple data segments is based on a combination of the first candidate set of likely sentence pauses and the second candidate set of likely sentence pauses.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein:
the identification of the first candidate set of likely sentence pauses comprises:
analyzing each data chunk of the multiple data chunks to identify acoustic features occurring within each corresponding chunk of speech audio; and
providing indications of the acoustic features occurring within each chunk of speech audio to the acoustic model as input; and
the identification of the candidate sets of words likely spoken in the speech audio comprises:
subdividing a data segment of the multiple data segments into multiple data frames that each represent a portion of a corresponding speech segment;
analyzing each data frame of the multiple data frames to identify acoustic features occurring within each corresponding portion of the corresponding speech segment; and
providing indications of the acoustic features occurring within each portion of the corresponding speech segment to the acoustic model as input.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein:
the acoustic model outputs probability distributions indicative of probabilities of occurrence of speech sounds for each of multiple specific times within a period of time; and
the identification of the candidate sets of words likely spoken in the speech audio comprises analyzing the probability distributions output by the acoustic model to identify combinations of speech sounds that correspond to words.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein, in response to the request, the at least one processor is caused to perform further speech-to-text operations comprising:
use a language model with the candidate sets of words to derive corresponding candidate sets of n-grams accompanied by corresponding indications of relative probabilities of use of each n-gram within each candidate set of n-grams; and
use at least one of probability distributions output by the acoustic model or indications of relative probabilities output by the language model to select each word included in the transcript from either a corresponding candidate set of words or a corresponding candidate set of n-grams.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the selection of each word included in the transcript is further based on a relative weighting of the acoustic model and the language model that is based on a level of audio noise of the speech audio.

### Claim 11

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:
receive, from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, the at least one processor is caused to perform pre-processing operations of a first segmentation technique comprising:
divide the speech data set into multiple data chunks that each represent a chunk of multiple chunks of the speech audio;
use an acoustic model with each data chunk of the multiple data chunks to identify pauses in the speech audio; and
analyze a length of time of each identified pause in the speech audio to identify a first candidate set of likely sentence pauses in the speech audio;
in response to the request, the at least one processor is caused to perform speech-to-text operations comprising:
based, at least in part, on the first candidate set of likely sentence pauses, divide the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
use the acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
analyze the identified likely speech sounds to identify candidate sets of words likely spoken in the speech audio;
based, at least in part, on the candidate sets of words likely spoken in the speech audio, generate a transcript of the speech data set; and
transmit an indication of the generation of the transcript to the requesting device.

### Claim 12 (depends on 11)

The computer-program product of claim 11, wherein:
the acoustic model is based on a neural network;
the neural network comprises a connectionist temporal classification (CTC) output;
the CTC output generates strings of one or more blank symbols for the identified pauses;
a quantity of blank symbols in each string of one or more blank symbols correlates to a length of a corresponding identified pause; and
the identification of the first candidate set of likely sentence pauses comprises:
comparing the length of each identified speech pause indicated by a corresponding string of one or more blank symbols to a predetermined blank threshold length; and
designating each identified speech pause that is at least as long as the predetermined blank threshold length as a likely sentence pause of the first candidate set of likely sentence pauses.

### Claim 13 (depends on 12)

The computer-program product of claim 12, wherein:
the acoustic model outputs indications of occurrences of speech sounds as indications of occurrences of graphemes of a set of graphemes;
the blank symbol comprises a grapheme of the set of graphemes that is indicative of a pause between speech sounds; and
the identification of candidate sets of words likely spoken in the speech audio comprises analyzing the indications of occurrence of graphemes output by the acoustic model to identify combinations of graphemes that correspond to words.

### Claim 14 (depends on 12)

The computer-program product of claim 12, wherein the at least one processor is caused to perform operations comprising:
prior to the use of the acoustic model to identify pauses in the speech audio, configure at least one neuromorphic device to instantiate at least one instance of the acoustic model; and
following the use of the acoustic model to identify pauses in the speech audio, maintain the configuration of the at least one neuromorphic device to maintain the at least one instance of the acoustic model to enable use of the at least one instance of the acoustic model to identify speech sounds in the speech audio.

### Claim 15 (depends on 12)

The computer-program product of claim 12, wherein:
the at least one processor comprises a processor of a control device of a distributed processing system, and multiple processors of multiple node devices of the distributed processing system; and
the processor of the control device is caused to distribute indications of a configuration of the neural network of the acoustic model to the multiple node devices to enable instantiation of multiple instances of the acoustic model within the multiple node devices by the multiple processors of the multiple node devices.

### Claim 16 (depends on 12)

The computer-program product of claim 12, wherein:
in response to the request, the at least one processor is caused to perform pre-processing operations of a second segmentation technique comprising:
divide the speech audio into multiple alternate data chunks that each represent an alternate chunk of multiple alternate chunks of the speech audio;
derive a threshold amplitude based on at least one peak amplitude of the speech audio;
compare a peak amplitude of the alternate chunk of the speech audio of each alternate data chunk to the threshold amplitude;
designate each alternate data chunk with a peak amplitude that is below the threshold amplitude as an alternate pause data chunk;
within at least one set of temporally consecutive alternate data chunks of the multiple alternate chunks, identify a longest subset of temporally consecutive alternate pause data chunks; and
for each set of the at least one set of temporally consecutive alternate data chunks, store an indication of the longest subset of temporally consecutive alternate pause data chunks as a likely sentence pause of a second candidate set of likely sentence pauses; and
the division of the of the speech data set into the multiple data segments is based on a combination of the first candidate set of likely sentence pauses and the second candidate set of likely sentence pauses.

### Claim 17 (depends on 11)

The computer-program product of claim 11, wherein:
the identification of the first candidate set of likely sentence pauses comprises:
analyzing each data chunk of the multiple data chunks to identify acoustic features occurring within each corresponding chunk of speech audio; and
providing indications of the acoustic features occurring within each chunk of speech audio to the acoustic model as input; and
the identification of the candidate sets of words likely spoken in the speech audio comprises:
subdividing a data segment of the multiple data segments into multiple data frames that each represent a portion of the corresponding speech segment;
analyzing each data frame of the multiple data frames to identify acoustic features occurring within each corresponding portion of the corresponding speech segment; and
providing indications of the acoustic features occurring within each portion of the corresponding speech segment to the acoustic model as input.

### Claim 18 (depends on 11)

The computer-program product of claim 11, wherein:
the acoustic model outputs probability distributions indicative of probabilities of occurrence of speech sounds for each of multiple specific times within a period of time; and
the identification of the candidate sets of words likely spoken in the speech audio comprises analyzing the probability distributions output by the acoustic model to identify combinations of speech sounds that correspond to words.

### Claim 19 (depends on 18)

The computer-program product of claim 18, wherein, in response to the request, the at least one processor is caused to perform further speech-to-text operations comprising:
use a language model with the candidate sets of words to derive corresponding candidate sets of n-grams accompanied by corresponding indications of relative probabilities of use of each n-gram within each candidate set of n-grams; and
use at least one of probability distributions output by the acoustic model or indications of relative probabilities output by the language model to select each word included in the transcript from either a corresponding candidate set of words or a corresponding candidate set of n-grams.

### Claim 20 (depends on 19)

The computer-program product of claim 19, wherein the selection of each word included in the transcript is further based on a relative weighting of the acoustic model and the language model that is based on a level of audio noise of the speech audio.

### Claim 21

A computer-implemented method comprising:
receiving, by at least one processor, and from a requesting device via a network, a request to perform speech-to-text conversion of a specified speech data set representing speech audio;
in response to the request, performing pre-processing operations of a first segmentation technique comprising:
dividing the speech data set into multiple data chunks that each represent a chunk of multiple chunks of the speech audio;
using, by the at least one processor, an acoustic model with each data chunk of the multiple data chunks to identify pauses in the speech audio; and
analyzing, by the at least one processor, a length of time of each identified pause in the speech audio to identify a first candidate set of likely sentence pauses in the speech audio;
in response to the request, performing speech-to-text operations comprising:
based, at least in part, on the first candidate set of likely sentence pauses, dividing the speech data set into multiple data segments that each represent a speech segment of multiple speech segments of the speech audio;
using, by the at least one processor, the acoustic model with each data segment of the multiple data segments to identify likely speech sounds in the speech audio;
analyzing, by the at least one processor, the identified likely speech sounds to identify candidate sets of words likely spoken in the speech audio;
based, at least in part, on the candidate sets of words likely spoken in the speech audio, generating, by the at least one processor, a transcript of the speech data set; and
transmitting, from the at least one processor, an indication of the generation of the transcript to the requesting device.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, wherein:
the acoustic model is based on a neural network;
the neural network comprises a connectionist temporal classification (CTC) output;
the CTC output generates strings of one or more blank symbols for the identified pauses;
a quantity of blank symbols in each string of one or more blank symbols correlates to a length of a corresponding identified pause; and
the identification of the first candidate set of likely sentence pauses comprises performing operations comprising:
comparing, by the at least one processor, the length of each identified speech pause indicated by a corresponding string of one or more blank symbols to a predetermined blank threshold length; and
designating, by the at least one processor, each identified speech pause that is at least as long as the predetermined blank threshold length as a likely sentence pause of the first candidate set of likely sentence pauses.

### Claim 23 (depends on 22)

The computer-implemented method of claim 22, wherein:
the acoustic model outputs indications of occurrences of speech sounds as indications of occurrences of graphemes of a set of graphemes;
the blank symbol comprises a grapheme of the set of graphemes that is indicative of a pause between speech sounds; and
the identification of candidate sets of words likely spoken in the speech audio comprises performing operations comprising analyzing, by the at least one processor, the indications of occurrence of graphemes output by the acoustic model to identify combinations of graphemes that correspond to words.

### Claim 24 (depends on 22)

The computer-implemented method of claim 22, comprising:
prior to the use of the acoustic model to identify pauses in the speech audio, configuring, by the at least one processor, at least one neuromorphic device to instantiate at least one instance of the acoustic model; and
following the use of the acoustic model to identify pauses in the speech audio, maintaining, by the at least one processor, the configuration of the at least one neuromorphic device to maintain the at least one instance of the acoustic model to enable use of the at least one instance of the acoustic model to identify speech sounds in the speech audio.

### Claim 25 (depends on 22)

The computer-implemented method of claim 22, wherein:
the at least one processor comprises a processor of a control device of a distributed processing system, and multiple processors of multiple node devices of the distributed processing system; and
the method comprises, distributing, by the processor of the control device, indications of a configuration of the neural network of the acoustic model to the multiple node devices to enable instantiation of multiple instances of the acoustic model within the multiple node devices by the multiple processors of the multiple node devices.

### Claim 26 (depends on 22)

The computer-implemented method of claim 22, wherein:
the method comprises, in response to the request, performing pre-processing operations of a second segmentation technique comprising:
dividing the speech audio into multiple alternate data chunks that each represent an alternate chunk of multiple alternate chunks of the speech audio;
deriving, by the at least one processor, a threshold amplitude based on at least one peak amplitude of the speech audio;
comparing, by the at least one processor, a peak amplitude of the alternate chunk of the speech audio of each alternate data chunk to the threshold amplitude;
designating, by the at least one processor, each alternate data chunk with a peak amplitude that is below the threshold amplitude as an alternate pause data chunk;
within at least one set of temporally consecutive alternate data chunks of the multiple alternate chunks, identifying, by the at least one processor, a longest subset of temporally consecutive alternate pause data chunks; and
for each set of the at least one set of temporally consecutive alternate data chunks, storing an indication of the longest subset of temporally consecutive alternate pause data chunks as a likely sentence pause of a second candidate set of likely sentence pauses; and
the division of the of the speech data set into the multiple data segments is based on a combination of the first candidate set of likely sentence pauses and the second candidate set of likely sentence pauses.

### Claim 27 (depends on 21)

The computer-implemented method of claim 21, wherein:
the identification of the first candidate set of likely sentence pauses comprises performing operations comprising:
analyzing, by the at least one processor, each data chunk of the multiple data chunks to identify acoustic features occurring within each corresponding chunk of speech audio; and
providing, by the at least one processor, indications of the acoustic features occurring within each chunk of speech audio to the acoustic model as input; and
the identification of the candidate sets of words likely spoken in the speech audio comprises performing operations comprising:
subdividing a data segment of the multiple data segments into multiple data frames that each represent a portion of the corresponding speech segment;
analyzing, by the at least one processor, each data frame of the multiple data frames to identify acoustic features occurring within each corresponding portion of the corresponding speech segment; and
providing, by the at least one processor, indications of the acoustic features occurring within each portion of the corresponding speech segment to the acoustic model as input.

### Claim 28 (depends on 21)

The computer-implemented method of claim 21, wherein:
the acoustic model outputs probability distributions indicative of probabilities of occurrence of speech sounds for each of multiple specific times within a period of time; and
the identification of the candidate sets of words likely spoken in the speech audio comprises performing operations comprising analyzing, by the at least one processor, the probability distributions output by the acoustic model to identify combinations of speech sounds that correspond to words.

### Claim 29 (depends on 28)

The computer-implemented method of claim 28, comprising, in response to the request, performing further speech-to-text operations comprising:
using, by the at least one processor, a language model with the candidate sets of words to derive corresponding candidate sets of n-grams accompanied by corresponding indications of relative probabilities of use of each n-gram within each candidate set of n-grams; and
using, by the at least one processor, at least one of probability distributions output by the acoustic model or indications of relative probabilities output by the language model to select each word included in the transcript from either a corresponding candidate set of words or a corresponding candidate set of n-grams.

### Claim 30 (depends on 29)

The computer-implemented method of claim 29, wherein the selection of each word included in the transcript is further based on a relative weighting of the acoustic model and the language model that is based on a level of audio noise of the speech audio.
