US 8,027,832 B2Granted

Efficient language identification

AssigneeMicrosoft Corporation

InventorsWilliam D. Ramsey, Patricia M. Schmid, Kevin R. Powell

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system and methods of language identification of natural language text are presented. The system includes stored expected character counts and variances for a list of characters found in a natural language. Expected character counts and variances are stored for multiple languages to be considered during language identification. At run-time, one or more languages are identified for a text sample based on comparing actual and expected character counts. The present methods can be combined with upstream analyzing of Unicode ranges for characters in the text sample to limit the number of languages considered. Further, n-gram methods can be used in downstream processing to select the most probable language from among the languages identified by the present system and methods.

Claims20

Dependent claims

A method of storing information to be used in identifying the natural language of text, the method comprising:

receiving text documents, each text document being written in a known natural language;
counting occurrences of each unique feature, for a plurality of unique features, in each text document for each language of a plurality of candidate languages to generate expected feature probability values for each unique feature;
storing said expected feature probability values for each unique feature for each of the plurality of candidate languages;
receiving a text sample written in an unidentified natural language;
counting a total number of features in the text sample written in the unidentified natural language;
determining actual occurrences of one or more of the plurality of unique features in the text sample written in the unidentified natural language;
determining actual feature counts for the one or more unique features based on the actual occurrences;
calculating probability values for the one or more unique features based on the total number of features of the text sample written in the unidentified natural language, the stored expected feature probability values for each of the one or more unique features for each of the plurality of candidate languages, and the actual feature counts for the one or more unique features;
calculating language scores for the plurality of candidate languages using a computer, said calculating comprising estimating a joint probability of seeing the one or more unique features in the text sample with the actual feature counts for the one or more unique features by multiplying the probability values calculated for the one or more unique features in the text sample;
and
identifying at least one language for the text sample from among the plurality of candidate languages based on the language scores.

8 more elements

A computer readable storage medium including instructions which, when implemented, cause a computer to perform language identification, the instructions comprising:

a first module adapted to access, for each of a plurality of natural languages, a feature list and expected probability values associated with each of the listed features, each feature in the feature list being a unique feature associated with the natural language;
and
a second module adapted to receive a text sample written in an unidentified natural language, count a total number of features in the text sample, and determine actual occurrences of each of a plurality of unique features in the feature lists in at least one window of characters in the text sample, the second module being further adapted to calculate probability values associated with each of the plurality of unique features given the total number of features of the text sample, stored expected probability values for each of the plurality of unique features and the actual occurrences of each of the plurality of unique features, the second module being further adapted to calculate language scores for each of the plurality of languages, said calculation comprising estimating a joint probability of seeing all of the plurality of unique features by multiplying the probability values associated with each of the plurality of unique features, and wherein the second module identifies at least one language for the text sample from among the plurality of natural languages based on the language scores.

Dependencies are inferred from the claim text.