# US 10,482,876 B2 · Hierarchical speech recognition decoder > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,876 B2 - Document ID: US10482876B2 - Status: Granted - Kind: B2 - Application number: 16/148,884 - Filed: 2018-10-01 - Published: 2019-01-31 - Granted: 2019-11-19 - Earliest claimed: 2016-04-18 - Assignees: Interactions LLC - Applicants: Interactions LLC - Inventors: Ethan Selfridge; Michael Johnston - Web page: https://patentgrep.com/patent/US10482876B2 ## CPC - G10L 15/197: Probabilistic grammars, e.g. word n-grams - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/063: {Training} - G10L 2015/0631: {Creating reference templates; Clustering} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A speech interpretation module interprets the audio of user utterances as sequences of words. To do so, the speech interpretation module parameterizes a literal corpus of expressions by identifying portions of the expressions that correspond to known concepts, and generates a parameterized statistical model from the resulting parameterized corpus. When speech is received the speech interpretation module uses a hierarchical speech recognition decoder that uses both the parameterized statistical model and language sub-models that specify how to recognize a sequence of words. The separation of the language sub-models from the statistical model beneficially reduces the size of the literal corpus needed for training, reduces the size of the resulting model, provides more fine-grained interpretation of concepts, and improves computational efficiency by allowing run-time incorporation of the language sub-models. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving, over a computer network, an utterance of a user, the utterance having been accepted from the user at a client device as spoken input; storing the utterance, the storing comprising identifying a plurality of sub-expressions by applying a parameterized statistical model that determines likely n-grams of literal word tokens and concept placeholders included in the utterance and storing each of the sub-expressions in the data structure as either: a set of literal word tokens representing the sub-expression, or a concept placeholder representing the sub-expression and providing an indication of a language sub-model; determining likely textual representations of the sub-expressions stored as concept placeholders by applying the indicated language sub-models to the sub-expressions; generating a user-specific textual interpretation of the utterance, the textual interpretation being a combination of the literal word tokens and the determined likely textual representations of the sub-expressions. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein applying a language sub-model to a sub-expression to determine a likely textual representation of the sub-expression comprises: identifying one or more additional language sub-models to apply to portions of the sub-expression; accessing the one or more additional language sub-models; and applying the accessed additional language sub-models to the associated portions of the sub-expression to determine likely textual representations of the portions of the sub-expression. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein applying the indicated language sub-model comprises compiling the indicated language sub-model at runtime. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein a language sub-model is explicitly specified based on rules of a logical grammar. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein a language sub-model is a statistical language model trained from an input corpus. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the input corpus is a parameterized speech recognition corpus. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, wherein a likely textual representation of the sub-expression is selected by the statistical language model from a set of possible textual representations based on a likelihood of each textual representation being a correct textual interpretation of the sub-expression, wherein each of the possible textual representations in the set includes a weighted value, the weighted values indicating relative probabilities that the possible textual representations will appear in a user utterance. ### Claim 8 A non-transitory computer-readable storage medium storing computer program instructions executable by a processor to perform operations comprising: receiving, over a computer network, an utterance of a user, the utterance having been accepted from the user at a client device as spoken input; storing the utterance, the storing comprising identifying a plurality of sub-expressions by applying a parameterized statistical model that determines likely n-grams of literal word tokens and concept placeholders included in the utterance and storing each of the sub-expressions in the data structure as either: a set of literal word tokens representing the sub-expression, or a concept placeholder representing the sub-expression and providing an indication of a language sub-model; determining likely textual representations of the sub-expressions stored as concept placeholders by applying the indicated language sub-models to the sub-expressions; generating a user-specific textual interpretation of the utterance, the textual interpretation being a combination of the literal word tokens and the determined likely textual representations of the sub-expressions. ### Claim 9 (depends on 8) The non-transitory computer-readable storage medium of claim 8, wherein applying a language sub-model to a sub-expression to determine a likely textual representation of the sub-expression comprises: identifying one or more additional language sub-models to apply to portions of the sub-expression; accessing the one or more additional language sub-models; and applying the accessed additional language sub-models to the associated portions of the sub-expression to determine likely textual representations of the portions of the sub-expression. ### Claim 10 (depends on 8) The non-transitory computer-readable storage medium of claim 8, wherein applying the indicated language sub-model comprises compiling the indicated language sub-model at runtime. ### Claim 11 (depends on 8) The non-transitory computer-readable storage medium of claim 8, wherein a language sub-model is explicitly specified based on rules of a logical grammar. ### Claim 12 (depends on 8) The non-transitory computer-readable storage medium of claim 8, wherein a language sub-model is a statistical language model trained from an input corpus. ### Claim 13 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein the input corpus is a parameterized speech recognition corpus. ### Claim 14 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein a likely textual representation of the sub-expression is selected by the statistical language model from a set of possible textual representations based on a likelihood of each textual representation being a correct textual interpretation of the sub-expression, wherein each of the possible textual representations in the set includes a weighted value, the weighted values indicating relative probabilities that the possible textual representations will appear in a user utterance. ### Claim 15 A system comprising: a processor for executing computer program instructions; and a non-transitory computer-readable storage medium storing computer program instructions executable by the processor to perform operations comprising: receiving, over a computer network, an utterance of a user, the utterance having been accepted from the user at a client device as spoken input; storing the utterance, the storing comprising identifying a plurality of sub-expressions by applying a parameterized statistical model that determines likely n-grams of literal word tokens and concept placeholders included in the utterance and storing each of the sub-expressions in the data structure as either: a set of literal word tokens representing the sub-expression, or a concept placeholder representing the sub-expression and providing an indication of a language sub-model; determining likely textual representations of the sub-expressions stored as concept placeholders by applying the indicated language sub-models to the sub-expressions; generating a user-specific textual interpretation of the utterance, the textual interpretation being a combination of the literal word tokens and the determined likely textual representations of the sub-expressions. ### Claim 16 (depends on 15) The system of claim 15, wherein applying a language sub-model to a sub-expression to determine a likely textual representation of the sub-expression comprises: identifying one or more additional language sub-models to apply to portions of the sub-expression; accessing the one or more additional language sub-models; and applying the accessed additional language sub-models to the associated portions of the sub-expression to determine likely textual representations of the portions of the sub-expression. ### Claim 17 (depends on 15) The system of claim 15, wherein applying the indicated language sub-model comprises compiling the indicated language sub-model at runtime. ### Claim 18 (depends on 15) The system of claim 15, wherein a language sub-model is explicitly specified based on rules of a logical grammar. ### Claim 19 (depends on 15) The system of claim 15, wherein a language sub-model is a statistical language model trained from an input corpus. ### Claim 20 (depends on 19) The computer-implemented method of claim 19, wherein a likely textual representation of the sub-expression is selected by the statistical language model from a set of possible textual representations based on a likelihood of each textual representation being a correct textual interpretation of the sub-expression, wherein each of the possible textual representations in the set includes a weighted value, the weighted values indicating relative probabilities that the possible textual representations will appear in a user utterance.