# US 2014/0207440 A1 · Language recognition based on vocabulary lists > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0207440 A1 - Document ID: US20140207440A1 - Status: Published application - Kind: A1 - Application number: 14/108,224 - Filed: 2013-12-16 - Published: 2014-07-24 - Earliest claimed: 2013-01-22 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Lu Li; Qiang Cheng; Jianxiong Ma; Feng Rao; Duling Lu; Li Lu; Xiang Zhang; Bo Chen - Web page: https://patentgrep.com/patent/US20140207440A1 ## CPC - G06F 40/242: Dictionaries - G06F 40/53: Processing of non-Latin text (kana-to-kanji conversion G06F40/129; vowelisation G06F40/232) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method is implemented at a computer to determine that certain information content is composed or compiled in a specific language selected among two or more similar languages. The computer integrates a first vocabulary list of a first language and a second vocabulary list of a second language into a comprehensive vocabulary list. The integrating includes analyzing the first vocabulary list in view of the second vocabulary list to identify a first vocabulary sub-list that is used in the first language, but not in the second language. The computer then identifies, in the information content, a plurality of expressions that are included in the comprehensive vocabulary list, and a subset of expressions that are included in the first vocabulary sub-list. Upon a determination that a total frequency of occurrence of the subset of expressions meets predetermined occurrence criteria, the computer determines that the information content is composed in the first language. ## Claims ### Claim 1 A computer-implemented method of recognizing a first language used in information content, comprising: at a computer having one or more processors and memory for storing programs to be executed by the one or more processors: integrating a first vocabulary list and a second vocabulary list that are built based on a first language and a second language, respectively, into a comprehensive vocabulary list, wherein the integrating includes analyzing the first vocabulary list in view of the second vocabulary list to at least identify a first vocabulary sub-list, in the comprehensive vocabulary list, that is used in the first language, but not in the second language; identifying, within the information content, a plurality of expressions that are included in the comprehensive vocabulary list; identifying, within the plurality of expressions, a subset of expressions that are included in the first vocabulary sub-list; determining that a total frequency of occurrence of the subset of expressions within the information content meets predetermined occurrence criteria; and in accordance with the determination, determining that the information content is composed in the first language. ### Claim 2 (depends on 1) The method of claim 1, wherein building the first vocabulary list and the second vocabulary list, further comprising: creating the respective vocabulary list for either one of the first language and the second language; identifying a plurality of test expressions in test information content corresponding to the respective language; after identifying, computing a respective probability value based on a frequency of occurrence for each of the plurality of test expressions; and when the respective probability value for each test expression exceeds a respective test threshold value, modifying the respective vocabulary list by incorporating the respective test expression. ### Claim 3 (depends on 2) The method of claim 2, wherein computing a respective probability value based on a frequency of occurrence for each of the plurality of test expressions, further comprising: determining a correlation value, a plurality of left entropy values and a plurality of right entropy values for the respective test expression in the plurality of test words, the correlation value being related to mutual information dependence among words in the respective test expression, each of the plurality of right entropy values being associated with a probability of coupling a respective right expression, in the respective language, to the right of the respective test expression, each of the plurality of left entropy values being associated with a probability of coupling a respective left expression, in the respective language, to the left of the respective test expression; and computing the respective probability value for the respective test expression by summing up the correlation value, the plurality of left entropy values and the plurality of right entropy values in a weighted manner. ### Claim 4 (depends on 1) The method of claim 1, wherein the subset of expressions are included in the first vocabulary sub-list, when each word in the subset of expressions is included in the first vocabulary list, but not in the second vocabulary list. ### Claim 5 (depends on 1) The method of claim 1, wherein determining that, in the information content, the total frequency of occurrence for the subset of expressions meets the predetermined occurrence criteria, based on at least one parameter among the number of words in the subset of expressions, a first ratio between the number of words in the subset of expressions and the total number of words in the information content, and a second ratio between the number of words in the subset of expressions and the number of words in another subset of expressions, included in the information content, that are defined as valid vocabulary. ### Claim 6 (depends on 1) The method of claim 1, wherein the total frequency of occurrence for the subset of expressions is determined to meet the predetermined occurrence criteria, when the number of words in the subset of expressions exceeds a threshold number. ### Claim 7 (depends on 1) The method of claim 1, wherein the total frequency of occurrence for the subset of expressions is determined to meet the predetermined occurrence criteria, when a ratio between the number of words in the subset of expressions and the total number of words in the information content exceed a first threshold ratio. ### Claim 8 (depends on 1) The method of claim 1, wherein the frequency of occurrence for the subset of expressions is determined to meet the predetermined occurrence criteria, when a ratio between the number of words in the subset of expressions and the number of words in another subset of expressions, included in the information content, that are defined as valid vocabulary exceeds a second threshold ratio. ### Claim 9 (depends on 1) The method of claim 1, wherein determining that, in the information content, the total frequency of occurrence for the subset of expressions meets the predetermined occurrence criteria, further comprising: computing the number of words in the subset of expressions that are specifically associated with the first language; computing a first ratio between the number of words in the subset of expressions and the total number of words in the information content; computing a second ratio between the number of words in the subset of expressions and the number of words in another subset of expressions, included in the information content, that are defined as valid vocabulary; calculating a weighted sum of the number of words in the subset of expressions, the first ratio and the second ratio; and determining that the weighted sum exceeds a predetermined threshold value. ### Claim 10 (depends on 1) The method of claim 1, wherein the first and second languages are two distinct Chinese dialects, and wherein identifying the plurality of expressions, within the information content, that are included in the comprehensive vocabulary list, further comprises: segmenting the information content to the plurality of expressions in accordance with the comprehensive vocabulary list. ### Claim 11 (depends on 1) The method of claim 1, wherein the first and second languages are two distinct languages selected from a language group that consists of British English, American English and Australian English. ### Claim 12 (depends on 1) The method of claim 1, wherein the second vocabulary list is analyzed in view of the first vocabulary list to identify a second vocabulary sub-list, in the comprehensive vocabulary list, that is used in the second language, but not in the first language, the subset of expressions in the plurality of expressions comprising a first subset of expressions, the total frequency of occurrence comprising a first frequency of occurrence, the method further comprising: identifying a second subset of expressions, in the plurality of expressions, that are specifically associated with the second language and included in the second vocabulary sub-list; determining, in the information content, a second frequency of occurrence for the second subset of expressions; and determining a primary language used by the information content according to a comparison between the first frequency of occurrence and the second frequency of occurrence. ### Claim 13 (depends on 1) The method of claim 1, wherein the first vocabulary list and the second vocabulary list are analyzed to identify a mixed vocabulary sub-list and a second vocabulary sub-list in the comprehensive vocabulary list, the mixed vocabulary sub-list being used in both the first and second languages, the second vocabulary sub-list being used in the second language, but not in the first language, the total frequency of occurrence comprising a first frequency of occurrence, the method further comprising: determining, in the information content, a second frequency of occurrence for expressions included in the second vocabulary sub-list; determining, in the information content, a mixed frequency of occurrence for expressions included in the mixed vocabulary sub-list; and determining a primary language used by the information content based on the first frequency of occurrence, the second frequency of occurrence and the mixed frequency of occurrence. ### Claim 14 A computer system, comprising: one or more processors; and memory having instructions stored thereon, which when executed by the one or more processors cause the processors to perform operations, the operations including: integrating a first vocabulary list and a second vocabulary list that are built based on a first language and a second language, respectively, into a comprehensive vocabulary list, wherein the integrating includes analyzing the first vocabulary list in view of the second vocabulary list to at least identify a first vocabulary sub-list, in the comprehensive vocabulary list, that is used in the first language, but not in the second language; identifying, within the information content, a plurality of expressions that are included in the comprehensive vocabulary list; identifying, within the plurality of expressions, a subset of expressions that are included in the first vocabulary sub-list; determining that a total frequency of occurrence of the subset of expressions within the information content meets predetermined occurrence criteria; and in accordance with the determination, determining that the information content is composed in the first language. ### Claim 15 (depends on 14) The computer system of claim 14, wherein building the first vocabulary list and the second vocabulary list, further comprising: creating the respective vocabulary list for either one of the first language and the second language; identifying a plurality of test expressions in test information content corresponding to the respective language; after identifying, computing a respective probability value based on a frequency of occurrence for each of the plurality of test expressions; and when the respective probability value for each test expression exceeds a respective test threshold value, modifying the respective vocabulary list by incorporating the respective test expression. ### Claim 16 (depends on 14) The computer system of claim 14, wherein determining that, in the information content, the total frequency of occurrence for the subset of expressions meets the predetermined occurrence criteria, based on at least one parameter among the number of words in the subset of expressions, a first ratio between the number of words in the subset of expressions and the total number of words in the information content, and a second ratio between the number of words in the subset of expressions and the number of words in another subset of expressions, included in the information content, that are defined as valid vocabulary. ### Claim 17 (depends on 14) The computer system of claim 14, wherein the first and second languages are two distinct Chinese dialects, and wherein identifying the plurality of expressions, within the information content, that are included in the comprehensive vocabulary list, further comprises: segmenting the information content to the plurality of expressions in accordance with the comprehensive vocabulary list. ### Claim 18 (depends on 14) The computer system of claim 14, wherein the first vocabulary list and the second vocabulary list are analyzed to identify a mixed vocabulary sub-list and a second vocabulary sub-list in the comprehensive vocabulary list, the mixed vocabulary sub-list being used in both the first and second languages, the second vocabulary sub-list being used in the second language, but not in the first language, the total frequency of occurrence comprising a first frequency of occurrence, the method further comprising: determining, in the information content, a second frequency of occurrence for expressions included in the second vocabulary sub-list; determining, in the information content, a mixed frequency of occurrence for expressions included in the mixed vocabulary sub-list; and determining a primary language used by the information content based on the first frequency of occurrence, the second frequency of occurrence and the mixed frequency of occurrence. ### Claim 19 A non-transitory computer-readable medium, having instructions stored thereon, which when executed by one or more processors cause the processors to perform operations, the operations including: integrating a first vocabulary list and a second vocabulary list that are built based on a first language and a second language, respectively, into a comprehensive vocabulary list, wherein the integrating includes analyzing the first vocabulary list in view of the second vocabulary list to at least identify a first vocabulary sub-list, in the comprehensive vocabulary list, that is used in the first language, but not in the second language; identifying, within the information content, a plurality of expressions that are included in the comprehensive vocabulary list; identifying, within the plurality of expressions, a subset of expressions that are included in the first vocabulary sub-list; determining that a total frequency of occurrence of the subset of expressions within the information content meets predetermined occurrence criteria; and in accordance with the determination, determining that the information content is composed in the first language. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein building the first vocabulary list and the second vocabulary list, further comprising: creating the respective vocabulary list for either one of the first language and the second language; identifying a plurality of test expressions in test information content corresponding to the respective language; after identifying, computing a respective probability value based on a frequency of occurrence for each of the plurality of test expressions; and when the respective probability value for each test expression exceeds a respective test threshold value, modifying the respective vocabulary list by incorporating the respective test expression.