# US 10,311,860 B2 · Language model biasing system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,860 B2 - Document ID: US10311860B2 - Status: Granted - Kind: B2 - Application number: 15/432,620 - Filed: 2017-02-14 - Published: 2018-08-16 - Granted: 2019-06-04 - Earliest claimed: 2017-02-14 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Petar Aleksic; Pedro J. Moreno Mengibar - Web page: https://patentgrep.com/patent/US10311860B2 ## CPC - G10L 15/07: to the speaker - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams - G10L 15/197: Probabilistic grammars, e.g. word n-grams - G10L 15/01: Assessment or evaluation of speech recognition systems - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Methods, systems, and apparatus for receiving audio data corresponding to a user utterance and context data, identifying an initial set of one or more n-grams from the context data, generating an expanded set of one or more n-grams based on the initial set of n-grams, adjusting a language model based at least on the expanded set of n-grams, determining one or more speech recognition candidates for at least a portion of the user utterance using the adjusted language model, adjusting a score for a particular speech recognition candidate determined to be included in the expanded set of n-grams, determining a transcription of user utterance that includes at least one of the one or more speech recognition candidates, and providing the transcription of the user utterance for output. ## Claims ### Claim 1 A computer-implemented method comprising: receiving audio data corresponding to a user utterance and context data for the user utterance; identifying, based on the context data, an initial set of one or more n-grams including one or more n-grams that do not represent speech preceding the user utterance; generating an expanded set of one or more n-grams based at least on the initial set of n-grams, the expanded set of n-grams comprising one or more n-grams that are different from the n-grams in the initial set of n-grams; based at least on the expanded set of n-grams, adjusting a language model trained to predict a first set of n-grams to be able to predict an additional n-gram in the expanded set of n-grams; determining one or more speech recognition candidates for at least a portion of the user utterance using the adjusted language model, wherein each speech recognition candidate comprises one or more words; after determining the one or more speech recognition candidates, adjusting a score for a particular speech recognition candidate based on determining that the particular speech recognition candidate is included in the expanded set of n-grams; after adjusting the score for the particular speech recognition candidate, determining, a transcription for the user utterance that includes at least one of the one or more speech recognition candidates; and providing the transcription of the user utterance for output. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein adjusting the language model based at least on the expanded set of n-grams comprises adjusting recognition scores for one or more of the n-grams of the expanded set of n-grams in the language model. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein adjusting the language model based at least on the expanded set of n-grams comprises adjusting probability masses assigned to one or more n-grams of the expanded set of n-grams in the language model. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the language model includes one or more placeholder transitions, and wherein adding the one or more n-grams of the expanded set of n-grams to the language model comprises: assigning a particular n-gram of the expanded set of n-grams to a particular placeholder transition. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein assigning the particular n-gram of the expanded set of n-grams to the particular placeholder transition comprises: adjusting a recognition score for the particular placeholder transition that is assigned the particular n-gram of the expanded set of n-grams. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein generating the expanded set of n-grams based at least on the initial set of n-grams comprises: sending one or more of the initial set of n-grams to each of one or more language expansion services; and receiving, from the one or more language expansion services and in response to sending the one or more of the initial set of n-grams to each of the one or more language expansion services, the expanded set of one or more n-grams. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein one or more of the expanded set of n-grams are included in a hash map, and wherein adjusting the score for the particular speech recognition candidate determined to be included in the expanded set of n-grams comprises: determining that the particular speech recognition candidate is included in the hash map. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the audio data corresponding to the user utterance corresponds to a particular segment of a spoken user input that comprises multiple segments; and wherein the language model is a general language model or a general language model that has been influenced during processing of a segment preceding the particular segment of the spoken user input. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the context data does not include one or more words included in a previous transcription of a user utterance. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the language model is adjusted based at least on the expanded set of n-grams prior to receiving the audio data corresponding to the user utterance. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein adjusting the language model is performed in response to receiving the context data for the user utterance, the context data indicating a context at a time the user utterance is spoken. ### Claim 12 (depends on 1) The method of claim 1, wherein receiving the audio data comprises receiving audio data detected by a user device; and wherein the initial set of n-grams comprises one or more words or phrases displayed on a screen of the user device. ### Claim 13 (depends on 1) The method of claim 1, wherein receiving the audio data comprises receiving audio data detected by a user device; and wherein one or more of the initial set of n-grams are provided by an application running on the user device. ### Claim 14 (depends on 1) The method of claim 1, wherein receiving the audio data comprises receiving audio data detected by a user device; and wherein the initial set of n-grams include a predetermined set of terms corresponding to a current dialog state corresponding one of multiple steps for carrying out a task. ### Claim 15 (depends on 1) The method of claim 1, wherein the context data includes an application identifier or a dialog state identifier; wherein the method comprises: retrieving a set of terms based on the application identifier or dialog state identifier; and including the retrieved set of terms in the expanded set of one or more terms. ### Claim 16 (depends on 1) The method of claim 1, wherein receiving the context data comprises receiving context data indicating a current context of a user device when the user device generates the audio data for the user utterance. ### Claim 17 A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving audio data corresponding to a user utterance and context data for the user utterance; identifying, based on the context data, an initial set of one or more n-grams including one or more n-grams that do not represent speech preceding the user utterance; generating an expanded set of one or more n-grams based at least on the initial set of n-grams, the expanded set of n-grams comprising one or more n-grams that are different from the n-grams in the initial set of n-grams; based at least on the expanded set of n-grams, adjusting a language model trained to predict a first set of n-grams to be able to predict an additional n-gram in the expanded set of n-grams; determining one or more speech recognition candidates for at least a portion of the user utterance using the adjusted language model, wherein each speech recognition candidate comprises one or more words; after determining the one or more speech recognition candidates, adjusting a score for a particular speech recognition candidate based on determining that the particular speech recognition candidate is included in the expanded set of n-grams; after adjusting the score for the particular speech recognition candidate, determining, a transcription for the user utterance that includes at least one of the one or more speech recognition candidates; and providing the transcription of the user utterance for output. ### Claim 18 (depends on 17) The system of claim 17, wherein adjusting the language model based at least on the expanded set of n-grams comprises adjusting the language model to generate scores for candidate transcriptions using adjusted probability scores different from probability scores determined through training of the language model, the adjusted probability scores indicating increased probabilities for one or more of the n-grams of the expanded set of n-grams compared to the probability scores determined through training of the language model. ### Claim 19 (depends on 17) The system of claim 17, wherein the language model includes one or more placeholder transitions, and wherein adding the one or more n-grams of the expanded set of n-grams to the language model comprises: assigning a particular n-gram of the expanded set of n-grams to a particular placeholder transition for determining the one or more speech recognition candidates for at least a portion of the user utterance; and wherein the operations comprise: after determining the one or more speech recognition candidates for at least a portion of the user utterance using the adjusted language model, removing the assignment of the particular n-gram to the particular placeholder transition. ### Claim 20 A non-transitory computer-readable storage device storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising: receiving audio data corresponding to a user utterance and context data for the user utterance; identifying an initial set of one or more n-grams from the context data; generating an expanded set of one or more n-grams based at least on the initial set of n-grams, the expanded set of n-grams comprising one or more n-grams that are different from the n-grams in the initial set of n-grams; adjusting a language model based at least on the expanded set of n-grams; determining one or more speech recognition candidates for at least a portion of the user utterance using the adjusted language model, wherein each speech recognition candidate comprises one or more words; after determining the one or more speech recognition candidates for at least a portion of the user utterance using the adjusted language model, undoing the adjustment to the language model; after determining the one or more speech recognition candidates, adjusting a score for a particular speech recognition candidate determined to be included in the expanded set of n-grams; after adjusting the score for the particular speech recognition candidate, determining, a transcription for the user utterance that includes at least one of the one or more speech recognition candidates; and providing the transcription of the user utterance for output.