Natural language detection

A system comprising:

• a memory;
• and
• a processing device operatively coupled to the memory, the processing device configured to:
• identify a document comprising a plurality of words in one or more natural languages;
• for each word of at least a subset of words of the document:
• generate a plurality of sets of tokens representing the word, wherein each set of tokens of the plurality of sets of tokens represents the word using a corresponding plurality of tokens defined for a corresponding natural language of a set of natural languages, and
• identify, based on the plurality of sets of tokens, a primary natural language associated with the word;
• associate each natural language of the set of natural languages with a corresponding word count indicating a number of words of the subset of words for which the natural language has been identified as the primary natural language;
• identify, among the set of natural languages, a natural language associated with a maximum word count;
• and
• associate the identified natural language with the document.

A non-transitory computer-readable storage medium including executable instructions that, when executed by a processing device, cause the processing device to:

• identify a document comprising a plurality of words in one or more natural languages;
• for each word of at least a subset of words of the document:
• generate a plurality of sets of tokens representing the word, wherein each set of tokens of the plurality of sets of tokens represents the word using a corresponding plurality of tokens defined for a corresponding natural language of a set of natural languages, and
• identify, based on the plurality of sets of tokens, a primary natural language associated with the word;
• associate each natural language of the set of natural languages with a corresponding word count indicating a number of words of the subset of words for which the natural language has been identified as the primary natural language;
• identify, among the set of natural languages, a natural language associated with a maximum word count;
• and
• associate the identified natural language with the document.