Automatic language identification using both N-gram and word information

A system for automatically identifying a natural language that is likely to be the predominant language of a sample text, the system comprising:

• text data defining the sample text;
• probability data for each of a set of natural languages;
• the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data;
• the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text;
• the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text;
• and
• a processor connected for accessing the text data and the probability data;
• the processor operating to:
• use the text data and the probability data to automatically obtain, for each of a third subset that includes at least two of the languages in the set, sample probability data indicating a probability that the sample text would occur in the language;
• for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data;
• for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data;
• and use the sample probability data to automatically obtain language identifying data;
• the language identifying data identifying the language whose sample probability data indicate the highest probability.

An article of manufacture for use in a system for automatically identifying a natural language that is likely to be the predominant language of a sample text;

• the system including:
• text data defining the sample text;
• a storage medium access device;
• and
• a processor connected for receiving data accessed on a storage medium by the storage medium access device and for accessing the text data;
• the article of manufacture comprising:
• a storage medium;
• probability data for each of a set of natural languages, stored by the storage medium;
• the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data;
• the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text;
• the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text;
• and
• instruction data stored by the storage medium;
• the instruction data indicating instructions the processor can execute;
• the processor, in executing the instructions:
• using the text data and the probability data to automatically obtain, for each of a third subset that includes at least two of the languages in the set, sample probability data indicating a probability that the sample text would occur in the language;
• for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data;
• for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data;
• and
• using the sample probability data to automatically obtain language identifying data;
• the language identifying data identifying the language whose sample probability data indicate the highest probability.

A method of operating a first machine to transfer data to a second machine over a network, the second machine including a memory and a processor connected for accessing the memory and for receiving text data defining a sample text;

• the method comprising:
• establishing a connection between the first and second machines over the network;
• and
• operating the first machine to transfer probability data for each of a set of natural languages and instruction data to the memory of the second machine;
• the probability data for each of a first subset that includes one or more of the languages including N-gram probability data and the probability data for each of a second subset that includes one or more of the languages including word probability data;
• the N-gram probability data for each language in the first subset indicating, for each of a set of N-grams, a probability that the N-gram occurs in a text if the language is the predominant language of the text;
• the word probability data of each language in the second subset indicating, for each of a set of words, a probability that the word occurs in a text if the language is the predominant language of the text;
• the instruction data indicating instructions the processor can execute;
• the processor, in executing the instructions, automatically identifying a natural language that is likely to be the predominant language of the sample text;
• the processor operating to:
• use the text data and the probability data to automatically obtain, for each of a third subset that includes at least two of the languages in the set, sample probability data indicating a probability that the sample text would occur in the language;
• for at least one language that is in the third subset and in the first subset, the sample probability data including information from the language's N-gram probability data;
• for at least one language that is in the third subset and in the second subset, the sample probability data including information from the language's word probability data;
• and
• use the sample probability data to automatically obtain language identifying data;
• the language identifying data identifying the language whose sample probability data indicate the highest probability.