# US 10,121,471 B2 · Language model speech endpointing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,471 B2 - Document ID: US10121471B2 - Status: Granted - Kind: B2 - Application number: 14/753,811 - Filed: 2015-06-29 - Published: 2016-12-29 - Granted: 2018-11-06 - Earliest claimed: 2015-06-29 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Bjorn Hoffmeister; Ariya Rastrow; Baiyang Liu - Web page: https://patentgrep.com/patent/US10121471B2 ## CPC - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 15/18: using natural language modelling - G10L 15/183: using context dependencies, e.g. language models - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 25/87: Detection of discrete points within a voice signal - G10L 25/93: Discriminating between voiced and unvoiced parts of speech signals (G10L25/90 takes precedence) - G10L 2025/783: {based on threshold decision} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract An automatic speech recognition (ASR) system detects an endpoint of an utterance using the active hypotheses under consideration by a decoder. The ASR system calculates the amount of non-speech detected by a plurality of hypotheses and weights the non-speech duration by the probability of each hypotheses. When the aggregate weighted non-speech exceeds a threshold, an endpoint may be declared. ## Claims ### Claim 1 A computer-implemented method for determining an endpoint during automatic speech recognition (ASR) processing, the method comprising: receiving audio data representing speech detected using a microphone of a mobile device; performing ASR processing on the audio data to determine a plurality of hypotheses; determining, for each of the plurality of hypotheses, a respective probability that the respective hypothesis corresponds to the audio data; determining, for each of the plurality of hypotheses, a respective number of non-speech audio frames immediately preceding a first point in the audio data; determining, for each of the plurality of hypotheses, a respective score by multiplying the probability of the respective hypothesis by a factor corresponding to the number of non-speech audio frames of the respective hypothesis; determining a cumulative score by summing the respective scores for each of the plurality of hypotheses; determining that the cumulative score exceeds a first threshold; and designating the first point as corresponding to a likely endpoint as a result of the cumulative score exceeding the first threshold. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: determining a second plurality of hypotheses, wherein: the second plurality of hypotheses is a subset of the plurality of hypotheses, and each of the second plurality of hypotheses is at a respective end state; determining a second score by summing the respective score for each of the second plurality of hypotheses; and determining that the second score exceeds a second threshold. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein determining the respective number of non-speech audio frames for each respective hypothesis comprises determining, for each hypothesis, a number of consecutive non-speech nodes immediately prior to the first point. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein determining the respective number of non-speech audio frames for each respective hypothesis comprises maintaining a counter, for each hypothesis, of a number of consecutive non-speech frames immediately prior to the first point. ### Claim 5 A computer-implemented method, comprising: receiving audio data; performing speech recognition processing on the audio data to determine a plurality of hypotheses, each hypothesis of the plurality comprising at least one of a representation of a respective subword unit or a representation of a respective word, wherein the plurality of hypotheses includes: a first hypothesis comprising a first representation of first non-speech preceding a first point in the audio data, and a second hypothesis comprising a second representation of second non-speech preceding the first point in the audio data; determining a first probability corresponding to the first hypothesis; determining a second probability corresponding to the second hypothesis; determining a first weighted duration value by using the first probability to adjust a first value representing a first time duration of the first non-speech; determining a second weighted duration value by using the second probability to adjust a second value representing a second time duration of the second non-speech; combining at least the first weighted duration value and the second weighted duration value to determine a third value representing an expected non-speech time duration preceding the first point in the audio data; determining that the third value exceeds a threshold duration value; and determining, based at least in part on determining that the third value exceeds the threshold duration value, that an endpoint of speech occurs at the first point in the audio data. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein: using the first probability to adjust the first value comprises multiplying the first value by the first probability; using the second probability to adjust the second value comprises multiplying the second value by the second probability, and combining the first weighted duration value and the second weighted duration value comprises summing the first weighted duration value and the second weighted duration value. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, further comprising: determining that the first hypothesis comprises a first indication of an end state; and determining that the second hypothesis comprises a second indication of an end state. ### Claim 8 (depends on 5) The computer-implemented method of claim 5, further comprising: determining the first time duration by determining a number of consecutive non-speech audio frames corresponding to the first representation of the first non-speech. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein determining the number of consecutive non-speech audio frames further comprises maintaining a counter of the number of consecutive non-speech audio frames corresponding to the first representation of the first non-speech. ### Claim 10 (depends on 8) The computer-implemented method of claim 8, wherein determining the number of consecutive non-speech audio frames further comprises determining a number of consecutive non-speech nodes represented in the first hypothesis. ### Claim 11 A computing system, comprising: at least one processor; a memory including instructions operable to be executed by the at least one processor to cause the computing system to perform a set of actions comprising: receiving audio data; performing speech recognition processing on the audio data to determine a plurality of hypotheses, each hypothesis of the plurality comprising at least one of a representation of a respective subword unit or a representation of a respective word, wherein the plurality of hypotheses includes: a first hypothesis comprising a first representation of first non-speech preceding a first point in the audio data, and a second hypothesis comprising a second representation of second non-speech preceding the first point in the audio data; determining a first probability corresponding to the first hypothesis; determining a second probability corresponding to the second hypothesis; determining a first weighted duration value by using the first probability to adjust a first value representing a first time duration of the first non-speech; determining a second weighted duration value by using the second probability to adjust a second value representing a second time duration of the second non-speech; combining at least the first weighted duration value and the second weighted duration value to determine a third value representing an expected non-speech time duration preceding the first point in the audio data; determining that the third value exceeds a threshold duration value; and determining, based at least in part on determining that the third value exceeds the threshold duration value, that an endpoint of speech occurs at the first point in the audio data. ### Claim 12 (depends on 11) The computing system of claim 11, wherein: using the first probability to adjust the first value comprises multiplying the first value by the first probability; using the second probability to adjust the second value comprises multiplying the second value by the second probability, and combining the first weighted duration value and the second weighted duration value comprises summing the first weighted duration value and the second weighted duration value. ### Claim 13 (depends on 11) The computing system of claim 11, wherein the memory includes additional instructions operable to be executed by the at least one processor to further cause the computing system to perform additional actions comprising: determining that the first hypothesis comprises a first indication of an end state; and determining that the second hypothesis comprises a second indication of an end state. ### Claim 14 (depends on 11) The computing system of claim 11, wherein the memory includes additional instructions operable to be executed by the at least one processor to further cause the computing system to perform additional actions comprising: determining the first time duration by determining a number of consecutive non-speech audio frames corresponding to the first representation of the first non-speech. ### Claim 15 (depends on 14) The computing system of claim 14, wherein determining the number of consecutive non-speech audio frames further comprises maintaining a counter of the number of consecutive non-speech audio frames corresponding to the first representation of the first non-speech. ### Claim 16 (depends on 14) The computing system of claim 14, wherein determining the number of consecutive non-speech audio frames further comprises determining a number of consecutive non-speech nodes represented in the first hypothesis. ### Claim 17 (depends on 7) The computer-implemented method of claim 7, further comprising: determining the first time duration by determining a number of consecutive non-speech audio frames corresponding to the first representation of the first duration of non-speech. ### Claim 18 (depends on 17) The computer-implemented method of claim 17, wherein determining the number of consecutive non-speech audio frames further comprises maintaining a counter of the number of consecutive non-speech audio frames corresponding to the first representation of the first non-speech. ### Claim 19 (depends on 17) The computer-implemented method of claim 17, wherein determining the number of non-speech audio frames further comprises determining a number of consecutive non-speech nodes represented in the first hypothesis. ### Claim 20 (depends on 13) The computing system of claim 13, wherein the memory includes additional instructions operable to be executed by the at least one processor to further cause the computing system to perform additional actions comprising: determining the first time duration by determining a number of consecutive non-speech audio frames corresponding to the first representation of the first non-speech. ### Claim 21 (depends on 20) The computing system of claim 20, wherein determining the number of consecutive non-speech audio frames further comprises maintaining a counter of the number of consecutive non-speech audio frames corresponding to the first representation of the first non-speech. ### Claim 22 (depends on 20) The computing system of claim 20, wherein determining the number of consecutive non-speech audio frames further comprises determining a number of consecutive non-speech nodes represented in the first hypothesis.