System and method for identifying language using morphologically-based techniques

An automatic language identification system for determining a source language of a text, comprising:

• a language corpus analyzer that generates, for each of a plurality of fixed-length suffixes extracted from at least one of a plurality of language corpuses each representing a natural language, a plurality of probabilities each associated with one suffix and one of said plurality of natural languages, each of said probabilities representing a relative likelihood that said associated source language is one of the natural languages based on a frequency with which said suffix occurs in the associated language corpus, and
• a language identification engine that determines, for each of said represented languages, an arithmetic sum of relative probabilities for each of a plurality of suffixes which appear in the text,
• wherein said relative frequency is normalized to correspond to a reference language corpus having a word count that is greater than that of said plurality of language corpuses, and
• wherein each said probability is equal to said normalized frequency for said corresponding language corpus divided by a total number of occurrences of said suffix in all language corpuses, and
• wherein said source language is identified as being one of said plurality of said represented languages having a greatest arithmetic sum of said relative probabilities.

An automatic language identification system for determining a source language of a text, said source language being one or a plurality of represented languages, the system comprising:

• first means for determining, for each of said plurality of represented languages, an arithmetic sum of relative probabilities for each of a plurality of word endings which appear in the text, the system indicating said source language being in one of said plurality of represented languages that has associated therewith a greatest arithmetic sum of said relative probabilities including,
• second means for generating a normalized frequency list for each of a plurality of language corpuses each of which is associated with one of said represented languages, each said normalized frequency list including a normalized value indicating a number of appearances of an extracted word ending in a corresponding one of said plurality of said language corpuses;
• and
• third means for generating a probability table containing said plurality of probabilities using said normalized frequency values in said normalized frequency lists.

A method for identifying the language of a text, the language being one of a plurality of languages represented by a language corpus, the method comprising the steps of:

• a) parsing a word from the language corpus;
• b) extracting all suffixes from said parsed words, said suffixes being a last three characters of a word;
• c) updating a suffix frequency list corresponding to the language corpus with said extracted suffix and a normalized frequency of occurrence of said extracted suffix in said corresponding language corpus;
• d) repeating said steps (a) through (c) for each of the plurality of language corpuses to result in a plurality of frequency lists, each associated with one language corpus;
• and
• e) creating a probability table containing probabilities derived from said normalized frequency of occurrence and representing a relative likelihood that the language of the text is one of the represented languages due to the presence of said suffix.

A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform method steps for identifying the language of a text, the language being one of a plurality of languages represented by a language corpus, said method steps comprising:

• a) parsing a word from the language corpus;
• b) extracting all suffixes from said parsed words, said suffixes being a last three characters of a word;
• c) updating a suffix frequency list corresponding to the language corpus with said extracted suffix and a normalized frequency of occurrence of said extracted suffix in said corresponding language corpus;
• d) repeating said steps (a) through (c) for each of the plurality of language corpuses to result in a plurality of frequency lists, each associated with one language corpus;
• and
• e) creating a probability table containing probabilities derived from said normalized frequency of occurrence and representing a relative likelihood that the language of the text is one of the represented languages due to the presence of said suffix.