# US 2018/0322867 A1 · Method and device for recognizing speech based on chinese-english mixed dictionary

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322867 A1
- Document ID: US20180322867A1
- Status: Published application
- Kind: A1
- Application number: 15/860,706
- Filed: 2018-01-03
- Published: 2018-11-08
- Earliest claimed: 2017-05-04
- Assignees: Baidu Online Network Technology (Beijing) Co., Ltd.
- Applicants: Baidu Online Network Technology (Beijing) Co., Ltd.
- Inventors: Xiangang LI; Xuewei ZHANG
- Web page: https://patentgrep.com/patent/US20180322867A1

## CPC

- G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
- G06F 40/242: Dictionaries
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/063: {Training}
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L 2015/0631: {Creating reference templates; Clustering}
- G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments of the present disclosure provide a method and a device for recognizing a speech based on a Chinese-English mixed dictionary. The method includes acquiring a Chinese-English mixed dictionary marked by an international phonetic alphabet, in which, the Chinese-English mixed dictionary includes a Chinese dictionary and an English dictionary revised by Chinglish; by taking the Chinese-English mixed dictionary as a training dictionary, taking a one-layer Convolutional Neural Network and a five-layer Long Short-Term Memory as a model, taking syllables or words as a target and taking a connectionist temporal classifier as a training criterion, training the model to obtain a trained CTC acoustic model; and performing a speech recognition on a Chinese-English mixed language based on the trained CTC acoustic model.

## Claims

### Claim 1

A method for recognizing a speech based on a Chinese-English mixed dictionary, comprising:
acquiring a Chinese-English mixed dictionary marked by an International Phonetic Alphabet IPA, wherein, the Chinese-English mixed dictionary comprises a Chinese dictionary and an English dictionary revised by Chinglish;
by taking the Chinese-English mixed dictionary as a training dictionary, taking a one-layer Convolutional Neural Network CNN and a five-layer Long Short-Term Memory LSTM as a model, taking syllables or words as a target and taking a Connectionist Temporal Classifier CTC as a training criterion, training the model to obtain a trained CTC acoustic model; and
performing a speech recognition on a Chinese-English mixed language based on the trained CTC acoustic model.

### Claim 2 (depends on 1)

The method according to claim 1, wherein acquiring the Chinese-English mixed dictionary marked by the IPA comprises:
acquiring a Chinese dictionary marked by the IPA and an English dictionary marked by the IPA;
acquiring audio training data, wherein, the audio training data comprises a plurality of Chinglish sentences;
acquiring English words in the plurality of Chinglish sentences and Chinese pronunciations corresponding to the English words; and
adding the English words and the Chinese pronunciations corresponding to the English words to the English dictionary, to obtain the English dictionary revised by Chinglish.

### Claim 3 (depends on 1)

The method according to claim 1, wherein acquiring the Chinese-English mixed dictionary marked by the IPA comprises:
acquiring a Chinese dictionary marked by the IPA and an English dictionary marked by the IPA;
acquiring audio training data, wherein, the audio training data comprises a plurality of Chinglish sentences;
performing a phoneme decoding and a matching file division on the plurality of Chinglish sentences based on the English dictionary marked by the IPA, to obtain English words in the plurality of Chinglish sentences and Chinese pronunciations corresponding to the English words; and
generating the English dictionary revised by Chinglish based on the English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the English words and the English dictionary marked by the IPA.

### Claim 4 (depends on 3)

The method according to claim 3, wherein performing the phoneme decoding and the matching file division on the plurality of Chinglish sentences based on the English dictionary marked by the IPA, to obtain the English words in the plurality of Chinglish sentences and the Chinese pronunciations corresponding to the English words comprises:
performing the phoneme decoding on the plurality of Chinglish sentences based on the English dictionary marked by the IPA, to find an optimal path, to obtain frame positions corresponding to phonemes in the plurality of Chinglish sentences;
acquiring a matching file corresponding to the plurality of Chinglish sentences, wherein, the matching file comprises positions of respective phonemes in the plurality of Chinglish sentences and phonemes corresponding to the English words; and
determining positions of respective the English words in the plurality of Chinglish sentences based on the matching file and the frame positions corresponding to the phonemes in the plurality of Chinglish sentences, to divide to obtain the English words in the plurality of Chinglish sentences and the Chinese pronunciations corresponding to the English words.

### Claim 5 (depends on 3)

The method according to claim 3, before generating the English dictionary revised by Chinglish based on the English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the English words and the English dictionary marked by the IPA, further comprising:
acquiring word frequencies of respective phonemes in the English word for each English word in the plurality of Chinglish sentences;
acquiring high frequency phonemes whose corresponding word frequencies are greater than a preset word frequency and high frequency English words comprising the high frequency phonemes;
generating the English dictionary revised by Chinglish based on the English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the English words and the English dictionary marked by the IPA comprises:
generating the English dictionary revised by Chinglish based on the high frequency English words in the plurality of Chinglish sentences, Chinese pronunciations corresponding to the high frequency English words and the English dictionary marked by the IPA.

### Claim 6 (depends on 3)

The method according to claim 3, after generating the English dictionary revised by Chinglish based on the English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the English words and the English dictionary marked by the IPA, further comprising:
performing the phoneme decoding and the matching file division on the plurality of Chinglish sentences based on the English dictionary revised by Chinglish, to obtain new English words in the plurality of Chinglish sentences and Chinese pronunciations corresponding to the new English words; and
updating the English dictionary revised by Chinglish based on the new English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the new English words and the English dictionary revised by Chinglish.

### Claim 7 (depends on 1)

The method according to claim 1, wherein, by taking the Chinese-English mixed dictionary as the training dictionary, taking the one-layer CNN and the five-layer LSTM as the model, taking syllables or words as the target and taking the CTC as the training criterion, training the model to obtain the trained CTC acoustic model comprises:
extracting feature points in a Chinglish sentence using a filter bank FBANK as inputs of the model, and training the model by taking the one-layer CNN and the five-layer LSTM as the model, taking a matching file corresponding to the Chinglish sentence as the target and taking a Cross Entropy CE as the training criterion, to obtain an initial model; and
training the initial model by taking the Chinese-English mixed dictionary as the training dictionary, taking the initial model as the model, taking the syllables or words as the target and taking the CTC as the training criterion, to obtain the trained CTC acoustic model.

### Claim 8

A device for recognizing a speech based on a Chinese-English mixed dictionary, comprising:
a memory;
a processor; and
computer programs stored in the memory and executable by the processor,
wherein when the processor executes the computer programs, a method for recognizing a speech based on a Chinese-English mixed dictionary is performed, the method comprising:
acquiring a Chinese-English mixed dictionary marked by an International Phonetic Alphabet IPA, wherein, the Chinese-English mixed dictionary comprises a Chinese dictionary and an English dictionary revised by Chinglish;
by taking the Chinese-English mixed dictionary as a training dictionary, taking a one-layer Convolutional Neural Network CNN and a five-layer Long Short-Term Memory LSTM as a model, taking syllables or words as a target and taking a Connectionist Temporal Classifier CTC as a training criterion, training the model to obtain a trained CTC acoustic model; and
performing a speech recognition on a Chinese-English mixed language based on the trained CTC acoustic model.

### Claim 9 (depends on 8)

The device according to claim 8, wherein acquiring the Chinese-English mixed dictionary marked by the IPA comprises:
acquiring a Chinese dictionary marked by the IPA and an English dictionary marked by the IPA;
acquiring audio training data, wherein, the audio training data comprises a plurality of Chinglish sentences;
acquiring English words in the plurality of Chinglish sentences and Chinese pronunciations corresponding to the English words; and
adding the English words and the Chinese pronunciations corresponding to the English words to the English dictionary, to obtain the English dictionary revised by Chinglish.

### Claim 10 (depends on 8)

The device according to claim 8, wherein acquiring the Chinese-English mixed dictionary marked by the IPA comprises:
acquiring a Chinese dictionary marked by the IPA and an English dictionary marked by the IPA;
acquiring audio training data, wherein, the audio training data comprises a plurality of Chinglish sentences;
performing a phoneme decoding and a matching file division on the plurality of Chinglish sentences based on the English dictionary marked by the IPA, to obtain English words in the plurality of Chinglish sentences and Chinese pronunciations corresponding to the English words; and
generating the English dictionary revised by Chinglish based on the English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the English words and the English dictionary marked by the IPA.

### Claim 11 (depends on 10)

The device according to claim 10, wherein the method further comprises:
performing the phoneme decoding on the plurality of Chinglish sentences based on the English dictionary marked by the IPA, to find an optimal path, to obtain frame positions corresponding to phonemes in the plurality of Chinglish sentences;
acquiring a matching file corresponding to the plurality of Chinglish sentences, wherein, the matching file comprises positions of respective phonemes in the plurality of Chinglish sentences and phonemes corresponding to the English words; and
determining positions of respective the English words in the plurality of Chinglish sentences based on the matching file and the frame positions corresponding to the phonemes in the plurality of Chinglish sentences, to divide to obtain the English words in the plurality of Chinglish sentences and the Chinese pronunciations corresponding to the English words.

### Claim 12 (depends on 10)

The device according to claim 10, wherein the method further comprises:
acquiring word frequencies of respective phonemes in the English word for each English word in the plurality of Chinglish sentences;
acquiring high frequency phonemes whose corresponding word frequencies are greater than a preset word frequency and high frequency English words comprising the high frequency phonemes;
generating the English dictionary revised by Chinglish based on the English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the English words and the English dictionary marked by the IPA comprises:
generating the English dictionary revised by Chinglish based on the high frequency English words in the plurality of Chinglish sentences, Chinese pronunciations corresponding to the high frequency English words and the English dictionary marked by the IPA.

### Claim 13 (depends on 10)

The device according to claim 10, wherein the method further comprises:
performing the phoneme decoding and the matching file division on the plurality of Chinglish sentences based on the English dictionary revised by Chinglish, to obtain new English words in the plurality of Chinglish sentences and Chinese pronunciations corresponding to the new English words; and
updating the English dictionary revised by Chinglish based on the new English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the new English words and the English dictionary revised by Chinglish.

### Claim 14 (depends on 8)

The device according to claim 8, wherein, by taking the Chinese-English mixed dictionary as the training dictionary, taking the one-layer CNN and the five-layer LSTM as the model, taking syllables or words as the target and taking the CTC as the training criterion, training the model to obtain the trained CTC acoustic model comprises:
extracting feature points in a Chinglish sentence using a filter bank FBANK as inputs of the model, and training the model by taking the one-layer CNN and the five-layer LSTM as the model, taking a matching file corresponding to the Chinglish sentence as the target and taking a Cross Entropy CE as the training criterion, to obtain an initial model; and
training the initial model by taking the Chinese-English mixed dictionary as the training dictionary, taking the initial model as the model, taking the syllables or words as the target and taking the CTC as the training criterion, to obtain the trained CTC acoustic model.

### Claim 15

A non-transitory computer readable storage medium, configured to store computer programs, wherein when the computer programs are executed by a processor, a method for recognizing a speech based on a Chinese-English mixed dictionary is performed, the method comprising:
acquiring a Chinese-English mixed dictionary marked by an International Phonetic Alphabet IPA, wherein, the Chinese-English mixed dictionary comprises a Chinese dictionary and an English dictionary revised by Chinglish;
by taking the Chinese-English mixed dictionary as a training dictionary, taking a one-layer Convolutional Neural Network CNN and a five-layer Long Short-Term Memory LSTM as a model, taking syllables or words as a target and taking a Connectionist Temporal Classifier CTC as a training criterion, training the model to obtain a trained CTC acoustic model; and
performing a speech recognition on a Chinese-English mixed language based on the trained CTC acoustic model.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium according to claim 15, wherein acquiring the Chinese-English mixed dictionary marked by the IPA comprises:
acquiring a Chinese dictionary marked by the IPA and an English dictionary marked by the IPA;
acquiring audio training data, wherein, the audio training data comprises a plurality of Chinglish sentences;
acquiring English words in the plurality of Chinglish sentences and Chinese pronunciations corresponding to the English words; and
adding the English words and the Chinese pronunciations corresponding to the English words to the English dictionary, to obtain the English dictionary revised by Chinglish.

### Claim 17 (depends on 15)

The non-transitory computer readable storage medium according to claim 15, wherein acquiring the Chinese-English mixed dictionary marked by the IPA comprises:
acquiring a Chinese dictionary marked by the IPA and an English dictionary marked by the IPA;
acquiring audio training data, wherein, the audio training data comprises a plurality of Chinglish sentences;
performing a phoneme decoding and a matching file division on the plurality of Chinglish sentences based on the English dictionary marked by the IPA, to obtain English words in the plurality of Chinglish sentences and Chinese pronunciations corresponding to the English words; and
generating the English dictionary revised by Chinglish based on the English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the English words and the English dictionary marked by the IPA.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium according to claim 17, wherein the method further comprises:
performing the phoneme decoding on the plurality of Chinglish sentences based on the English dictionary marked by the IPA, to find an optimal path, to obtain frame positions corresponding to phonemes in the plurality of Chinglish sentences;
acquiring a matching file corresponding to the plurality of Chinglish sentences, wherein, the matching file comprises positions of respective phonemes in the plurality of Chinglish sentences and phonemes corresponding to the English words; and
determining positions of respective the English words in the plurality of Chinglish sentences based on the matching file and the frame positions corresponding to the phonemes in the plurality of Chinglish sentences, to divide to obtain the English words in the plurality of Chinglish sentences and the Chinese pronunciations corresponding to the English words.

### Claim 19 (depends on 17)

The non-transitory computer readable storage medium according to claim 17, wherein the method further comprises:
acquiring word frequencies of respective phonemes in the English word for each English word in the plurality of Chinglish sentences;
acquiring high frequency phonemes whose corresponding word frequencies are greater than a preset word frequency and high frequency English words comprising the high frequency phonemes;
generating the English dictionary revised by Chinglish based on the English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the English words and the English dictionary marked by the IPA comprises:
generating the English dictionary revised by Chinglish based on the high frequency English words in the plurality of Chinglish sentences, Chinese pronunciations corresponding to the high frequency English words and the English dictionary marked by the IPA.

### Claim 20 (depends on 17)

The non-transitory computer readable storage medium according to claim 17, wherein the method further comprises:
performing the phoneme decoding and the matching file division on the plurality of Chinglish sentences based on the English dictionary revised by Chinglish, to obtain new English words in the plurality of Chinglish sentences and Chinese pronunciations corresponding to the new English words; and
updating the English dictionary revised by Chinglish based on the new English words in the plurality of Chinglish sentences, the Chinese pronunciations corresponding to the new English words and the English dictionary revised by Chinglish.
