# US 12,008,986 B1 · Universal semi-word model for vocabulary contraction in automatic speech recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,986 B1 - Document ID: US12008986B1 - Status: Granted - Kind: B1 - Application number: 16/859,938 - Filed: 2020-04-27 - Granted: 2024-06-11 - Earliest claimed: 2020-04-27 - Assignees: Interactions LLC - Applicants: Interactions LLC - Inventors: Ilija Zeljkovic; Andrej Ljolje - Web page: https://patentgrep.com/patent/US12008986B1 ## CPC - G10L 15/063: {Training} - G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/04: Segmentation; Word boundary detection - G10L 15/16: using artificial neural networks - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A speech recognition system includes, or has access to, conventional speech recognizer data, including a conventional acoustic model and pronunciation dictionary. The speech recognition system generates restructured speech recognizer data from the conventional speech recognizer data. When used at runtime by a speech recognizer module, the restructured speech recognizer data produces more accurate and efficient results than those produced using the conventional speech recognizer data. The restructuring involves segmenting entries of the conventional pronunciation dictionary and acoustic model according to their constituent phonemes and grouping those entries with the same initial N phonemes, for some integer N (e.g., N=3), and deriving a restructured dictionary with a corresponding semi-word acoustic model for the various grouped entries. The decomposition of the conventional pronunciation dictionary into the restructured dictionary with semi-word acoustic model greatly reduces the number of possibilities in the dictionaries (e.g., from potentially unlimited to finite and relatively small), and also improves the accuracy of speech recognition. ## Claims ### Claim 1 A computer-implemented method of a speech recognition system for recognizing a natural language utterance, the method comprising: accessing a pronunciation dictionary that maps words in an associated vocabulary to corresponding sequences of phonemes, and an acoustic model representing sound characteristics of phonemes in a language associated with the pronunciation dictionary; for a given predetermined number of phonemes, segmenting the words of the pronunciation dictionary into (i) an initial sequence of the predetermined number of phonemes, and (ii) a reference to a semi-word model representing a sequence of remaining phonemes in whole words that follow the predetermined number of phonemes; creating a restructured pronunciation dictionary based on the segmented initial sequences of phonemes and references to semi-word models representing sequences of remaining phonemes, the restructured pronunciation dictionary containing references to a semi-word model for words having more than the predetermined number of phonemes; using a multi-level neural network to train a restructured acoustic model for the acoustic model based on the segmented initial sequences of phonemes and references to semi-word models; receiving a natural language utterance; segmenting the natural language utterance into words at least in part using the restructured acoustic model; and for at least one of the words: recognizing, using the restructured pronunciation dictionary, a first sequence of the predetermined number of phonemes, determining, based on a presence of a reference to a semi-word model in the restructured pronunciation dictionary, that there are additional phonemes to be recognized in the word following the sequence of the predetermined number of phonemes, and responsive to determining that there are additional phonemes to be recognized in the word following the sequence of the predetermined number of phonemes, using at least one semi-word model for a part of the word to recognize text of the word. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: for a second one of the words: determining that an entry in the restructured pronunciation dictionary corresponding to phonemes of the word does not refer to a semi-word model; and responsive to the determining, recognizing the text of the word without using a semi-word model. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising determining any number of combinations of phonemes followed by a semi-word, responsive to recognizing the text using a semi-word model. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising predetermining the predetermined number of phonemes for the restructured pronunciation dictionary based on a number of words in the pronunciation dictionary. ### Claim 5 A non-transitory computer-readable storage medium storing instructions that when executed by a computer processor perform actions comprising: accessing a pronunciation dictionary that maps words in an associated vocabulary to corresponding sequences of phonemes, and an acoustic model representing sound characteristics of phonemes in a language associated with the pronunciation dictionary; for a given predetermined number of phonemes, segmenting the words of the pronunciation dictionary into (i) an initial sequence of the predetermined number of phonemes, and (ii) a reference to a semi-word model representing a sequence of remaining phonemes in whole words that follow the predetermined number of phonemes; creating a restructured pronunciation dictionary based on the segmented initial sequences of phonemes and references to semi-word models representing sequences of remaining phonemes, the restructured pronunciation dictionary containing references to a semi-word model for words having more than the predetermined number of phonemes; using a multi-level neural network to train a restructured acoustic model for the acoustic model based on the segmented initial sequences of phonemes and references to semi-word models; receiving a natural language utterance; segmenting the natural language utterance into words at least in part using the restructured acoustic model; and for at least one of the words: recognizing, using the restructured pronunciation dictionary, a first sequence of the predetermined number of phonemes in the word, determining, based on a presence of a reference to a semi-word model in the restructured pronunciation dictionary, that there are additional phonemes to be recognized in the word following the sequence of the predetermined number of phonemes, and responsive to determining that there are additional phonemes to be recognized in the word following the sequence of the predetermined number of phonemes, using at least one semi-word model for a remainder of the word comprising the additional phonemes to recognize text of the word. ### Claim 6 (depends on 5) The non-transitory computer-readable storage medium of claim 5, the actions further comprising: for a second one of the words: determining that an entry in the restructured pronunciation dictionary corresponding to phonemes of the word does not refer to a semi-word model; and responsive to the determining, recognizing the text of the word without using a semi-word model. ### Claim 7 (depends on 5) The non-transitory computer-readable storage medium of claim 5, the actions further comprising determining any number of combinations of phonemes followed by a semi-word, responsive to recognizing the text using a semi-word model. ### Claim 8 (depends on 5) The non-transitory computer-readable storage medium of claim 5, the actions further comprising predetermining the predetermined number of phonemes for the restructured pronunciation dictionary based on a number of words in the pronunciation dictionary. ### Claim 9 A speech recognition system comprising: a computer processor; and a non-transitory computer-readable storage medium storing instructions that when executed by the computer processor perform actions comprising: accessing a pronunciation dictionary that maps words in an associated vocabulary to corresponding sequences of phonemes, and an acoustic model representing sound characteristics of phonemes in a language associated with the pronunciation dictionary; for a given predetermined number of phonemes, segmenting the words of the pronunciation dictionary into (i) an initial sequence of the predetermined number of phonemes, and (ii) a reference to a semi-word model representing a sequence of remaining phonemes in whole words that follow the predetermined number of phonemes; creating a restructured pronunciation dictionary based on the segmented initial sequences of phonemes and references to semi-word models representing sequences of remaining phonemes, the restructured pronunciation dictionary containing references to a semi-word model for words having more than the predetermined number of phonemes; using a multi-level neural network to train a restructured acoustic model for the acoustic model based on the segmented initial sequences of phonemes and references to semi-word models; receiving a natural language utterance; segmenting the natural language utterance into words at least in part using the restructured acoustic model; and for at least one of the words: recognizing, using the restructured pronunciation dictionary, a first sequence of the predetermined number of phonemes, determining, based on a presence of a reference to a semi-word model in the restructured pronunciation dictionary, that there are additional phonemes to be recognized in the word following the sequence of the predetermined number of phonemes, and responsive to determining that there are additional phonemes to be recognized in the word following the sequence of the predetermined number of phonemes, using at least one semi-word model for a part of the word to recognize text of the word. ### Claim 10 (depends on 9) The speech recognition system of claim 9, the actions further comprising: for a second one of the words: determining that an entry in the restructured pronunciation dictionary corresponding to phonemes of the word does not refer to a semi-word model; and responsive to the determining, recognizing the text of the word without using a semi-word model. ### Claim 11 (depends on 9) The speech recognition system of claim 9, the actions further comprising determining any number of combinations of phonemes followed by a semi-word, responsive to recognizing the text using a semi-word model. ### Claim 12 (depends on 9) The speech recognition system of claim 9, the actions further comprising predetermining the predetermined number of phonemes for the restructured pronunciation dictionary based on a number of words in the pronunciation dictionary. ### Claim 13 A computer-implemented method of a speech recognition system for recognizing a natural language utterance, the method comprising: accessing a pronunciation dictionary that maps words in an associated vocabulary to corresponding sequences of phonemes, and an acoustic model representing sound characteristics of phonemes in a language associated with the pronunciation dictionary; for a given predetermined number of phonemes, segmenting the words of the pronunciation dictionary into (i) an initial sequence of the predetermined number of phonemes, and (ii) a reference to a semi-word model representing a sequence of remaining phonemes in whole words that follow the predetermined number of phonemes; creating a restructured pronunciation dictionary based on the segmented initial sequences of phonemes and references to semi-word models representing sequences of remaining phonemes, the restructured pronunciation dictionary containing references to a semi-word model for words having more than the predetermined number of phonemes; using a multi-level neural network to train a restructured acoustic model for the acoustic model based on the segmented initial sequences of phonemes and references to semi-word models; receiving a natural language utterance; segmenting the natural language utterance into words at least in part using the restructured acoustic model; and for at least one of the words: recognizing, using the restructured pronunciation dictionary, a first sequence of the predetermined number of phonemes within the word, determining, based on a presence of a reference to a semi-word model in the restructured pronunciation dictionary, that there are additional phonemes to be recognized in the word following the sequence of the predetermined number of phonemes, and responsive to determining that there are additional phonemes to be recognized in the word following the sequence of the predetermined number of phonemes, using at least one semi-word model for phonemes of the word, other than the phonemes of the first sequence, to recognize text of the word.