US 11,056,098 B1Granted

Silent phonemes for tracking end of speech

AssigneeAmazon Technologies, Inc.

InventorPushkaraksha Gejji

Life of this patent

Estimating the term…

Loading patent…

Abstract

Embodiments describe a method for speech endpoint detection including receiving identification data for a first state associated with a first frame of speech data from a WFST language model, determining that the first frame of the speech data includes silence data, incrementing a silence counter associated with the first state, copying a value of the silence counter of the first state to a corresponding silence counter field in a second state associated with the first state in an active state list, and determining that the value of the silence counter for the first state is above a silence threshold. The method further includes, determining that an endpoint of the speech has occurred in response to determining that the silence counter is above the silence threshold, and outputting text data representing a plurality of words determined from the speech data that was received prior to the endpoint.

Claims20

Dependent claims

A method for tracking end of speech, the method comprising:

receiving, by a fetch module implemented within a weighted finite state transducer (WFST) accelerator configured as a hardware component, identification data for a first state associated with a first frame of speech data from a WFST language model, the identification data including a silence bit that indicates whether the first frame includes audio data or silence data, wherein the WFST language model includes information about relationships between sounds and words for a spoken language;
determining, by an endpointing module implemented in the WFST accelerator that identifies an endpoint in the speech data, that the first frame of the speech data includes silence data;
incrementing, by the endpointing module, a silence counter associated with the first state, the silence counter being stored in an active state list within local memory of the WFST accelerator, wherein the active state list associates the first state with a second state, the second state representing a second frame of speech data different from the first frame of speech data represented by the first state;
copying, by the endpointing module, a value of the silence counter of the first state to a corresponding silence counter field in the second state;
determining, by the WFST accelerator, that the value of the silence counter for the first state is above a silence threshold, which indicates that a pause in speech has extended longer than a predetermined amount of time;
in response to determining that the silence counter is above the silence threshold, determining, by the WFST accelerator, that an endpoint of the speech has occurred;
and
outputting, by the WFST accelerator, text data representing a plurality of words determined from the speech data that was received prior to the endpoint.

5 more elements

An automatic speech recognition device for tracking end of speech, the automatic speech recognition device comprising:

a memory device storing a weighted finite state transducer (WFST) language model that includes information about relationships between a sound and a corresponding word of a spoken language;
a WFST accelerator implemented as a hardware component and comprising local memory, a first module, and a second module, wherein:
the first module receives first data for a first state associated with a first frame of speech data from the WFST language model, the first data including a silence bit that indicates whether the first frame of the speech data includes audio data or silence data;
the second module:
determines that the first frame includes silence data;
and
increments a first counter value associated with the first state;
and
the WFST accelerator:
determines that the first counter value is above a first threshold, wherein the first threshold is indicative of a predetermined amount of time that differentiates between a pause in speech and the end of speech;
and
determines that an endpoint of the speech has occurred;
and
a natural language understanding (NLU) module that receives a plurality of words determined from the speech data that was received prior to the endpoint from the WFST accelerator and determines an intent using the plurality of words.

11 more elements

An automatic speech recognition system for tracking end of speech, the automatic speech recognition system comprising:

an input/output (I/O) device interface coupled to:
a microphone configured to receive speech input and convert it into speech data that is a digital representation of the speech input;
an antenna configured to couple the I/O device interface to a server through a network;
and
a speaker configured to output sound to a user;
a memory device storing a weighted finite state transducer (WFST) language model that includes information about relationships between a sound and a corresponding word of a spoken language;
an audio processor coupled to the I/O device interface and configured to segment the speech data into frames;
a WFST accelerator implemented as a hardware component and comprising local memory, a first module, and a second module, wherein:
the first module receives first data for a first state associated with a first frame of speech data from the WFST language model, the first data including a silence bit that indicates whether the first frame of the speech data includes audio data or silence data;
the second module:
determines that the first frame includes silence data;
and
increments a first counter value associated with the first state;
and
the WFST accelerator:
determines that the first counter value is above a first threshold, wherein the first threshold is indicative of a predetermined amount of time that differentiates between a pause in speech and the end of speech;
and
determines that an endpoint of the speech has occurred;
and
a natural language understanding (NLU) module that receives a plurality of words determined from the speech data that was received prior to the endpoint from the WFST accelerator and determines an intent using the plurality of words.

17 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/455,381Filed 27 Jun 2019 · us provisional application
Priority: 28 Nov 2018
Granted: 6 Jul 2021
Term ends: Not estimatedThe source records don’t support an estimate.

Classification28 codes

G10L 15/05
Segmentation; Word boundary detection: word boundary detection
G06F 5/01
Methods or arrangements for data conversion without changing the order or content of the data handled: for shifting, e.g. justifying, scaling,…
G06F 7/5443
For evaluating functions by calculation: sum of products
G06F 7/57
Using non-contact-making devices, e.g. tube, solid state device; using unspecified devices: arithmetic logic units [ALU], i.e. arrangements or…