# US 2014/0236600 A1 · Method and device for keyword detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0236600 A1
- Document ID: US20140236600A1
- Status: Published application
- Kind: A1
- Application number: 14/103,775
- Filed: 2013-12-11
- Published: 2014-08-21
- Earliest claimed: 2013-01-29
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Li LU; Xiang ZHANG; Shuai YUE; Feng RAO; Eryu WANG; Lu LI
- Web page: https://patentgrep.com/patent/US20140236600A1

## CPC

- G10L 15/063: {Training}
- G10L 15/08: Speech classification or search
- G10L 2015/088: {Word spotting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

An electronic device with one or more processors and memory trains an acoustic model with an international phonetic alphabet (IPA) phoneme mapping collection and audio samples in different languages, where the acoustic model includes: a foreground model; and a background model. The device generates a phone decoder based on the trained acoustic model. The device collects keyword audio samples, decodes the keyword audio samples with the phone decoder to generate phoneme sequence candidates, and selects a keyword phoneme sequence from the phoneme sequence candidates. After obtaining the keyword phoneme sequence, the device detects one or more keywords in an input audio signal with the trained acoustic model, including: matching phonemic keyword portions of the input audio signal with phonemes in the keyword phoneme sequence with the foreground model; and filtering out phonemic non-keyword portions of the input audio signal with the background model.

## Claims

### Claim 1

A method of detecting keywords, comprising:
at an electronic device with one or more processors and memory:
training an acoustic model with an International Phonetic Alphabet (IPA) phoneme mapping collection and a plurality of audio samples in a plurality of different languages, wherein the acoustic model includes:
a foreground model configured to match a phoneme in an input audio signal to a corresponding keyword; and
a background model configured to match a phoneme in the input audio signal to a corresponding non-keyword;
after training the acoustic model, generating a phone decoder based on the trained acoustic model;
obtaining a keyword phoneme sequence, including:
collecting a set of keyword audio samples;
decoding the set of keyword audio samples with the phone decoder to generate a set of phoneme sequence candidates; and
selecting the keyword phoneme sequence from the set of phoneme sequence candidates;
after obtaining the keyword phoneme sequence, detecting one or more keywords in the input audio signal with the trained acoustic model, including:
matching one or more phonemic keyword portions of the input audio signal with one or more phonemes in the keyword phoneme sequence with the foreground model; and
filtering out one or more phonemic non-keyword portions of the input audio signal with the background model.

### Claim 2 (depends on 1)

The method of claim 1, wherein selecting the keyword phoneme sequence from the set of phoneme sequence candidates includes:
in accordance with a determination that the set of keyword audio samples includes one collected keyword audio sample, selecting the phoneme sequence candidate generated from decoding the one collected keyword audio sample as the keyword phoneme sequence; and
in accordance with a determination that the set of audio samples includes two or more collected keyword audio samples, selecting one of the two or more phoneme sequence candidates generated from decoding the two or more collected keyword audio samples as the keyword phoneme sequence.

### Claim 3 (depends on 1)

The method of claim 1, further including:
collecting the plurality of audio samples in the plurality of different languages and labeled data for the plurality of audio samples;
obtaining a phoneme collection for each of the plurality of different languages;
mapping phonemes from each phoneme collection to phonemes in the IPA so as to generate the IPA phoneme mapping collection; and
wherein the acoustic model is trained based on the collected plurality of audio samples in the plurality of different languages, the collected labeled data for the plurality of audio samples, and the generated IPA phoneme mapping collection.

### Claim 4 (depends on 3)

The method of claim 3, further including:
processing the collected plurality of audio samples with a predetermined characteristic extraction protocol so as to obtain a plurality of corresponding audio characteristic sequences;
obtaining a characteristic phoneme collection corresponding to the plurality of audio characteristic sequences based on the IPA phoneme mapping collection;
training the foreground and background models based on the characteristic phoneme collection and the collected labeled data; and
integrating the trained foreground and background models into the acoustic model.

### Claim 5 (depends on 4)

The method of claim 4, wherein training the foreground model includes:
for each phoneme in the characteristic phoneme collection:
training an initial hidden Markov model (HMM) for three statuses of a respective phoneme in the characteristic phoneme collection;
obtaining data related to the respective phoneme from the collected labeled data;
updating the initial HMM with the obtained data so as to obtain a monophone model for the respective phoneme;
for each phoneme in a phoneme collection corresponding to the labeled data:
obtaining a context phoneme; and
generating a triphone, by linking the context phoneme to the corresponding monophone;
clustering a status of the triphone with a decision tree;
performing Gaussian splitting training on the clustered triphone so as to update a parameter of the clustered triphone; and
after performing the Gaussian splitting training, performing minimum phoneme error discriminative training so as to obtain a triphone model for a respective phoneme in a phoneme collection corresponding to the labeled data; and
training the foreground model based on the obtained monophone and triphone models.

### Claim 6 (depends on 5)

The method of claim 5, further including:
calculating a Gaussian Mixed Model (GMM) distance between two monophone models;
comparing the calculated GMM distance with a predefined similarity threshold value; and
in accordance with a determination that the calculated GMM distance is larger than the predefined similarity threshold value:
clustering the two monophones corresponding to the two monophone models; and
recording the two monophones in a confusion matrix, wherein the confusion matrix is configured to describe similar monophones.

### Claim 7 (depends on 6)

The method of claim 6, wherein training the background model includes:
generating a confusion phoneme collection by processing the phonemes in the characteristic phoneme collection with the confusion matrix; and
training the background model with the generated confusion phoneme collection.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the set of audio samples includes two or more collected keyword audio samples; and
the set of phoneme sequence candidates generated from decoding the two or more collected keyword audio samples include two or more phoneme sequence candidates; and
the method including:
integrating the phoneme sequence candidates into a linear structure, including:
mapping the corresponding phonemes of the phoneme sequence candidates to an edge of the linear structure; and
categorizing the edges corresponding to similar phonemes into a same slot of the linear structure, wherein the slots form a linear connection relation with each other; and
selecting a path of the linear structure, wherein the phonemes corresponding to the edges of the selected path comprise the keyword phoneme sequence.

### Claim 9 (depends on 8)

The method of claim 8, wherein selecting the path from the linear structure includes:
calculating an occurrence frequency for the phoneme corresponding to each edge of the linear structure;
calculating a score for each path of the linear structure by summing the occurrence frequencies of the phonemes corresponding to each edge in a respective path;
sorting the scores for the paths of the linear structure from high to low, and selecting the first N paths as alternative paths, wherein N is an integer larger than 1; and
calculating a confidence measure for each of the N alternative paths, and selecting one of the N alternative paths according to corresponding calculated confidences measures, wherein the phonemes corresponding to the edges of the selected one of the N alternative paths comprise the keyword phoneme sequence.

### Claim 10 (depends on 9)

The method of claim 9, wherein calculating the confidence measure for each of the N alternative paths includes:
for each of the N alternative paths:
aligning a respective alternative path with each of the phoneme sequence candidates and calculating an intermediate confidence measure; and
after aligning the respective alternative path with each of the phoneme sequence candidates, calculating the average of the intermediate confidence measures, wherein the average of the calculated intermediate confidence measures is the confidence measure for the alternative path; and
wherein selecting one of the N alternative paths includes selecting the alternative path with the highest confidence measure, wherein the phonemes corresponding to the edges of the selected the alternative path with the highest confidence measure comprise the keyword phoneme sequence.

### Claim 11

An electronic device, comprising:
one or more processors; and
memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for:
training an acoustic model with an International Phonetic Alphabet (IPA) phoneme mapping collection and a plurality of audio samples in a plurality of different languages, wherein the acoustic model includes:
a foreground model configured to match a phoneme in an input audio signal to a corresponding keyword; and
a background model configured to match a phoneme in the input audio signal to a corresponding non-keyword;
after training the acoustic model, generating a phone decoder based on the trained acoustic model;
obtaining a keyword phoneme sequence, including:
collecting a set of keyword audio samples;
decoding the set of keyword audio samples with the phone decoder to generate a set of phoneme sequence candidates; and
selecting the keyword phoneme sequence from the set of phoneme sequence candidates;
after obtaining the keyword phoneme sequence, detecting one or more keywords in the input audio signal with the trained acoustic model, including:
matching one or more phonemic keyword portions of the input audio signal with one or more phonemes in the keyword phoneme sequence with the foreground model; and
filtering out one or more phonemic non-keyword portions of the input audio signal with the background model.

### Claim 12 (depends on 11)

The device of claim 11, wherein selecting the keyword phoneme sequence from the set of phoneme sequence candidates includes:
in accordance with a determination that the set of keyword audio samples includes one collected keyword audio sample, selecting the phoneme sequence candidate generated from decoding the one collected keyword audio sample as the keyword phoneme sequence; and
in accordance with a determination that the set of audio samples includes two or more collected keyword audio samples, selecting one of the two or more phoneme sequence candidates generated from decoding the two or more collected keyword audio samples as the keyword phoneme sequence.

### Claim 13 (depends on 11)

The device of claim 11, wherein the one or more programs comprising instructions for:
collecting the plurality of audio samples in the plurality of different languages and labeled data for the plurality of audio samples;
obtaining a phoneme collection for each of the plurality of different languages;
mapping phonemes from each phoneme collection to phonemes in the IPA so as to generate the IPA phoneme mapping collection; and
wherein the acoustic model is trained based on the collected plurality of audio samples in the plurality of different languages, the collected labeled data for the plurality of audio samples, and the generated IPA phoneme mapping collection.

### Claim 14 (depends on 13)

The device of claim 13, wherein the one or more programs comprising instructions for:
processing the collected plurality of audio samples with a predetermined characteristic extraction protocol so as to obtain a plurality of corresponding audio characteristic sequences;
obtaining a characteristic phoneme collection corresponding to the plurality of audio characteristic sequences based on the IPA phoneme mapping collection;
training the foreground and background models based on the characteristic phoneme collection and the collected labeled data; and
integrating the trained foreground and background models into the acoustic model.

### Claim 15 (depends on 11)

The device of claim 11, wherein:
the set of audio samples includes two or more collected keyword audio samples; and
the set of phoneme sequence candidates generated from decoding the two or more collected keyword audio samples include two or more phoneme sequence candidates; and
the method including:
integrating the phoneme sequence candidates into a linear structure, including:
mapping the corresponding phonemes of the phoneme sequence candidates to an edge of the linear structure; and
categorizing the edges corresponding to similar phonemes into a same slot of the linear structure, wherein the slots form a linear connection relation with each other; and
selecting a path of the linear structure, wherein the phonemes corresponding to the edges of the selected path comprise the keyword phoneme sequence.

### Claim 16

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device with one or more processors, cause the device to:
train an acoustic model with an International Phonetic Alphabet (IPA) phoneme mapping collection and a plurality of audio samples in a plurality of different languages, wherein the acoustic model includes:
a foreground model configured to match a phoneme in an input audio signal to a corresponding keyword; and
a background model configured to match a phoneme in the input audio signal to a corresponding non-keyword;
after training the acoustic model, generate a phone decoder based on the trained acoustic model;
obtain a keyword phoneme sequence, wherein the obtaining includes:
collecting a set of keyword audio samples;
decoding the set of keyword audio samples with the phone decoder to generate a set of phoneme sequence candidates; and
selecting the keyword phoneme sequence from the set of phoneme sequence candidates;
after obtaining the keyword phoneme sequence, detect one or more keywords in the input audio signal with the trained acoustic model, wherein the detecting includes:
matching one or more phonemic keyword portions of the input audio signal with one or more phonemes in the keyword phoneme sequence with the foreground model; and
filtering out one or more phonemic non-keyword portions of the input audio signal with the background model.

### Claim 17 (depends on 16)

The non-transitory computer readable storage medium of claim 16, wherein selecting the keyword phoneme sequence from the set of phoneme sequence candidates includes:
in accordance with a determination that the set of keyword audio samples includes one collected keyword audio sample, selecting the phoneme sequence candidate generated from decoding the one collected keyword audio sample as the keyword phoneme sequence; and
in accordance with a determination that the set of audio samples includes two or more collected keyword audio samples, selecting one of the two or more phoneme sequence candidates generated from decoding the two or more collected keyword audio samples as the keyword phoneme sequence.

### Claim 18 (depends on 16)

The non-transitory computer readable storage medium of claim 16, wherein the one or more programs comprising instructions, which further cause the device to:
collect the plurality of audio samples in the plurality of different languages and labeled data for the plurality of audio samples;
obtain a phoneme collection for each of the plurality of different languages;
map phonemes from each phoneme collection to phonemes in the IPA so as to generate the IPA phoneme mapping collection; and
wherein the acoustic model is trained based on the collected plurality of audio samples in the plurality of different languages, the collected labeled data for the plurality of audio samples, and the generated IPA phoneme mapping collection.

### Claim 19 (depends on 18)

The non-transitory computer readable storage medium of claim 18, wherein the one or more programs comprising instructions, which further cause the device to:
processing the collected plurality of audio samples with a predetermined characteristic extraction protocol so as to obtain a plurality of corresponding audio characteristic sequences;
obtaining a characteristic phoneme collection corresponding to the plurality of audio characteristic sequences based on the IPA phoneme mapping collection;
training the foreground and background models based on the characteristic phoneme collection and the collected labeled data; and
integrating the trained foreground and background models into the acoustic model.

### Claim 20 (depends on 16)

The non-transitory computer readable storage medium of claim 16, wherein:
the set of audio samples includes two or more collected keyword audio samples;
the set of phoneme sequence candidates generated from decoding the two or more collected keyword audio samples include two or more phoneme sequence candidates; and
the one or more programs comprising instructions, which further cause the device to:
integrate the phoneme sequence candidates into a linear structure, wherein the integrating includes:
mapping the corresponding phonemes of the phoneme sequence candidates to an edge of the linear structure; and
categorizing the edges corresponding to similar phonemes into a same slot of the linear structure, wherein the slots form a linear connection relation with each other; and
select a path of the linear structure, wherein the phonemes corresponding to the edges of the selected path comprise the keyword phoneme sequence.
