# US 10,311,878 B2 · Incorporating an exogenous large-vocabulary model into rule-based speech recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,878 B2 - Document ID: US10311878B2 - Status: Granted - Kind: B2 - Application number: 15/426,640 - Filed: 2017-02-07 - Published: 2017-06-08 - Granted: 2019-06-04 - Earliest claimed: 2014-01-17 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Travis Wilson; Salman Quazi; John Vicondoa; Pradip Fatehpuria - Web page: https://patentgrep.com/patent/US10311878B2 ## CPC - G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems - G10L 15/06: Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice (G10L15/14 takes precedence) - G10L 15/18: using natural language modelling - G10L 15/197: Probabilistic grammars, e.g. word n-grams - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 15/193: Formal grammars, e.g. finite state automata, context free grammars or word networks - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Incorporation of an exogenous large-vocabulary model into rule-based speech recognition is provided. An audio stream is received by a local small-vocabulary rule-based speech recognition system (SVSRS), and is streamed to a large-vocabulary statistically-modeled speech recognition system (LVSRS). The SVSRS and LVSRS perform recognitions of the audio. If a portion of the audio is not recognized by the SVSRS, a rule is triggered that inserts a mark-up in the recognition result. The recognition result is sent to the LVSRS. If a mark-up is detected, recognition of a specified portion of the audio is performed. The LVSRS result is unified with the SVSRS result and sent as a hybrid response back to the SVSRS. If the hybrid-recognition rule is not triggered, an arbitration algorithm is evoked to determine whether the SVSRS or the LVSRS recognition has a lesser word error rate. The determined recognition is sent as a response to the SVSRS. ## Claims ### Claim 1 A method for providing incorporation of an exogenous large-vocabulary model into rule-based speech recognition, comprising: receiving a first recognition result from a rule-based speech recognition system, the first recognition result including a mark-up that specifies a portion of a received audio stream was not recognized by the rule-based speech recognition system; performing a statistical model-based recognition of the marked-up portion of the audio stream to create a second recognition result; combining the second recognition result with the first recognition result to create a combined recognition result; and sending the combined recognition result to the rule-based speech recognition system. ### Claim 2 (depends on 1) The method of claim 1, wherein the second recognition result is a statistical model-based recognition result. ### Claim 3 (depends on 1) The method of claim 1, wherein combining the second recognition result with the first recognition result comprises replacing the mark-up specifying that the portion of the audio stream is not recognized by the rule-based speech recognition system with the second recognition result. ### Claim 4 (depends on 1) The method of claim 1, further comprising performing a statistical model-based recognition of the audio stream prior to receiving the first recognition result. ### Claim 5 (depends on 4) The method of claim 4, wherein if the first recognition result comprises a mark-up specifying that the portion of the audio stream is not recognized by the rule-based speech recognition system: cancelling performing the statistical model-based recognition of the audio stream; and performing a statistical model-based recognition of the specified portion of the audio stream. ### Claim 6 (depends on 1) The method of claim 1, further comprising: if the first recognition result does not comprise a mark-up specifying that the portion of the audio stream is not recognized by a rule-based speech recognition system: performing a statistical model-based recognition of the audio stream; analyzing a result of the statistical model-based recognition of the audio stream and the first recognition result; determining whether the result of the statistical model-based recognition of the audio stream or whether the first recognition result has a better recognition quality; and sending the recognition result with the better recognition quality to the rule-based speech recognition system. ### Claim 7 A system for providing speech recognition, comprising: one or more processors; and a memory coupled to the one or more processors, the one or more processors operable to: receive a first recognition result for a received audio stream, the first recognition results being received from a rule-based speech recognition system; determine if the first recognition result comprises a mark-up that indicates a portion of the audio stream was not recognized by the rule-based speech recognition system; when it is determined the first recognition result comprises the mark-up, performing a statistical model-based recognition of the marked-up portion of the audio stream to create a second recognition result; combine the second recognition result with the first recognition result to create a combined recognition result; and send the combined recognition result to the rule-based speech recognition system. ### Claim 8 (depends on 7) The system of claim 7, wherein the second recognition is a statistical model-based recognition result. ### Claim 9 (depends on 7) The system of claim 7, wherein the one or more processors are further operable to replace the mark-up specifying that the portion of the audio stream is not recognized by the rule-based speech recognition system with the second recognition result. ### Claim 10 (depends on 7) The system of claim 7, wherein the one or more processors are further operable to perform a statistical model-based recognition of the audio stream prior to receiving the first recognition result. ### Claim 11 (depends on 10) The system of claim 10, wherein if the first recognition result comprises a mark-up specifying that the portion of the audio stream is not recognized by the rule-based speech recognition system, the one or more processors are further operable to: cancel performing the statistical model-based recognition of the audio stream; and perform a statistical model-based recognition of the specified portion of the audio stream. ### Claim 12 (depends on 7) The system of claim 7, further comprising: if the first recognition result does not comprise a mark-up specifying that the portion of the audio stream is not recognized by the rule-based speech recognition system, the one or more processors are further operable to: perform a statistical model-based recognition of the audio stream; analyze a result of the statistical model-based recognition of the audio stream and the first recognition result; determine whether the result of the statistical model-based recognition of the audio stream or whether the first recognition result has a better recognition quality; and send the recognition result with the better recognition quality to the rule-based speech recognition system. ### Claim 13 A computer-readable storage device encoding computer executable instructions that, when executed by a processing unit, perform a method, comprising: receiving a first recognition result of an audio stream from a rule-based speech recognition system, the first recognition result containing a mark-up that indicates a portion of the audio stream was not recognized by the rule-based speech recognition system; performing a statistical model-based recognition of the marked-up portion of the audio stream to create a second recognition result; combining the second recognition result with the first recognition result to form a combined recognition result; and sending the combined recognition result to the rule-based speech recognition system. ### Claim 14 (depends on 13) The computer-readable storage device of claim 13, wherein the second recognition result is a statistical model-based recognition result. ### Claim 15 (depends on 13) The computer-readable storage device of claim 13, wherein combining the second recognition result with the first recognition result comprises replacing the mark-up portion of the audio stream with the second recognition result. ### Claim 16 (depends on 13) The computer-readable storage device of claim 13, further comprising performing a statistical model-based recognition of the audio stream prior to receiving the first recognition result. ### Claim 17 (depends on 16) The computer-readable storage device of claim 16, wherein if the first recognition result comprises a mark-up specifying that the portion of the audio stream is not recognized by the rule-based speech recognition system: cancelling performing the statistical model-based recognition of the audio stream; and performing a statistical model-based recognition of the specified portion of the audio stream. ### Claim 18 (depends on 14) The computer-readable storage device of claim 14, further comprising instructions for: performing a statistical model-based recognition of the audio stream; analyzing a result of the statistical model-based recognition of the audio stream and the first recognition result; determining whether the result of the statistical model-based recognition of the audio stream or whether the first recognition result has a better recognition quality; and sending the recognition result with the better recognition quality to the rule-based speech recognition system when it is determined that the first recognition result does not comprise a mark-up specifying that the portion of the audio stream is not recognized by a rule-based speech recognition system. ### Claim 19 (depends on 13) The computer-readable storage device of claim 13, further comprising instructions for performing a task based on the combined recognition result.