# US 2017/0287474 A1 · Improving Automatic Speech Recognition of Multilingual Named Entities

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0287474 A1
- Document ID: US20170287474A1
- Status: Published application
- Kind: A1
- Application number: 15/512,281
- Filed: 2014-09-26
- Published: 2017-10-05
- Earliest claimed: 2014-09-26
- Assignees: Nuance Communications, Inc.
- Applicants: Nuance Communications, Inc.
- Inventors: Paul MAERGNER; Paul VOZILA; Stefan HAHN; Nathan BODENSTAB
- Web page: https://patentgrep.com/patent/US20170287474A1

## CPC

- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
- G06F 40/242: Dictionaries
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/295: Named entity recognition
- G06F 40/45: Example-based machine translation; Alignment
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 15/063: {Training}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and systems are provided for improving speech recognition of multilingual named entities. In some embodiments, a list comprising a plurality of named entities may be accessed by a computing device. A first named entity represented in the native language may be compared with the first named entity represented in the foreign language. One or more words that appear in both the first named entity represented in the native language and the first named entity represented in the foreign language may be identified as one or more foreign words. A grapheme-to-phoneme (G2P) conversion may be applied to the one or more foreign words, wherein graphemes of the one or more foreign words are mapped to phonemes in the native language. The G2P conversion may result in a native pronunciation for each of the one or more foreign words, which are added to a recognition dictionary along with the native pronunciations.

## Claims

### Claim 1

A method comprising:
accessing, by a computing device, a list comprising a plurality of named entities, wherein each named entity is represented in a native language and a foreign language;
comparing a first named entity represented in the native language with the first named entity represented in the foreign language;
identifying one or more words that appear in both the first named entity represented in the native language and the first named entity represented in the foreign language, wherein the one or more words are identified as one or more foreign words; and
applying a grapheme-to-phoneme conversion to the one or more foreign words, resulting in a native pronunciation for each of the one or more foreign words.

### Claim 2 (depends on 1)

The method of claim 1, wherein applying the grapheme-to-phoneme conversion to the one or more foreign words further comprises:
mapping phonemes of foreign words in a foreign language dictionary to phonemes in the native language, resulting in a mapped native pronunciation for each of the foreign words in the foreign language dictionary using the phonemes in the native language;
training a grapheme-to-phoneme converter based on the mapped native pronunciation for each of the foreign words in the foreign language dictionary; and
mapping graphemes of the one or more foreign words to phonemes in the native language based on the grapheme-to-phoneme converter, resulting in the native pronunciation for each of the one or more foreign words.

### Claim 3 (depends on 1)

The method of claim 1, wherein accessing the list comprising the plurality of named entities further comprises:
determining a subset of named entities that match at least one predefined criterion; and
removing the subset of named entities from the list comprising the plurality of named entities.

### Claim 4 (depends on 3)

The method of claim 3, wherein the at least one predefined criterion is based on at least one of meta information, string similarity metric, or a language identification.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining a subset of the one or more foreign words that match at least one predefined metric; and
removing the subset of the one or more foreign words from the one or more identified foreign words.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
adding the one or more foreign words and the native pronunciation for each of the one or more foreign words to a recognition dictionary.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
adding the one or more foreign words as a token to the recognition dictionary, wherein the token indicates a single word or a multi-word phrase.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
receiving, by the computing device, a natural language input comprising the one or more foreign words; and
performing speech recognition of the natural language input by identifying the one or more foreign words as the token added to the recognition dictionary.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
accessing the list comprising the plurality of named entities in a multilingual database.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
obtaining a first list comprising named entities in the native language and a second list comprising named entities in the foreign language;
comparing the first list and the second list by string similarity analysis; and
compiling the list comprising the plurality of named entities based on the string similarity analysis results, wherein each named entity is represented in the native language and the foreign language.

### Claim 11 (depends on 10)

The method of claim 10, wherein the string similarity analysis comprises at least one of edit distance analysis, longest common substring analysis, or another string metric.

### Claim 12

A system comprising:
at least one processor; and
at least one memory storing computer-readable instructions that, when executed by the at least one processor, cause the system to:
access a list comprising a plurality of named entities, wherein each named entity is represented in a native language and a foreign language;
compare a first named entity represented in the native language with the first named entity represented in the foreign language;
identify one or more words that appear in both the first named entity represented in the native language and the first named entity represented in the foreign language, wherein the one or more words are identified as one or more foreign words; and
apply a grapheme-to-phoneme conversion to the one or more foreign words, resulting in a native pronunciation for each of the one or more foreign words.

### Claim 13 (depends on 12)

The system of claim 12, wherein the computer-readable instructions, when executed, cause the system to:
map phonemes of foreign words in a foreign language dictionary to phonemes in the native language, resulting in a mapped native pronunciation for each of the foreign words in the foreign language dictionary using the phonemes in the native language;
train a grapheme-to-phoneme converter based on the mapped native pronunciation for each of the foreign words in the foreign language dictionary; and
map graphemes of the one or more foreign words to phonemes in the native language based on the grapheme-to-phoneme converter, resulting in the native pronunciation for each of the one or more foreign words.

### Claim 14 (depends on 12)

The system of claim 12, wherein the computer-readable instructions, when executed, cause the system to:
add the one or more foreign words and the native pronunciation for each of the one or more foreign words to a recognition dictionary.

### Claim 15

One or more non-transitory computer-readable media having instructions stored thereon, which when performed, cause a computing device to perform:
accessing a list comprising a plurality of named entities, wherein each named entity is represented in a native language and a foreign language;
comparing a first named entity represented in the native language with the first named entity represented in the foreign language;
identifying one or more words that appear in both the first named entity represented in the native language and the first named entity represented in the foreign language, wherein the one or more words are identified as one or more foreign words; and
applying a grapheme-to-phoneme conversion to the one or more foreign words, resulting in a native pronunciation for each of the one or more foreign words.

### Claim 16 (depends on 15)

The non-transitory computer-readable media of claim 15, wherein applying the grapheme-to-phoneme conversion to the one or more foreign words comprises:
mapping phonemes of foreign words in a foreign language dictionary to phonemes in the native language, resulting in a mapped native pronunciation for each of the foreign words in the foreign language dictionary using the phonemes in the native language;
training a grapheme-to-phoneme converter based on the mapped native pronunciation for each of the foreign words in the foreign language dictionary; and
mapping graphemes of the one or more foreign words to phonemes in the native language based on the grapheme-to-phoneme converter, resulting in the native pronunciation for each of the one or more foreign words.

### Claim 17 (depends on 15)

The one or more non-transitory computer-readable media of claim 15, wherein the instructions further cause the computing device to perform:
adding the one or more foreign words and the native pronunciation for each of the one or more foreign words to a recognition dictionary.

### Claim 18 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein the instructions further cause the computing device to perform:
adding the one or more foreign words as a token to the recognition dictionary, wherein the token indicates a single word or a multi-word phrase.

### Claim 19 (depends on 15)

The one or more non-transitory computer-readable media of claim 15, wherein the instructions further cause the computing device to perform:
obtaining a first list comprising named entities in the native language and a second list comprising named entities in the foreign language;
comparing the first list and the second list by string similarity analysis; and
compiling the list comprising the plurality of named entities based on the string similarity analysis results, wherein each named entity is represented in the native language and the foreign language.

### Claim 20 (depends on 19)

The one or more non-transitory computer-readable media of claim 19, wherein the string similarity analysis comprises at least one of edit distance analysis, longest common substring analysis, or another string metric.