Multi-language relevance-based indexing and search

A computer-implemented method of indexing terms in multiple languages, comprising:

• segmenting information to be indexed into at least one token;
• determining, using at least one processor of at least one computing device, a language type for each token;
• analyzing, using the at least one processor of the at least one computing device, a language dictionary for the determined language type of each token and further segmenting the information into at least one additional token where appropriate based upon contents of the language dictionary;
• using the at least one processor of the at least one computing device, searching for synonyms in at least one of the multiple languages corresponding to the contents of each token after analyzing the language dictionary for each token;
• and
• storing the synonyms and contents of each token for use in indexing the information.

A system for indexing terms in multiple languages, comprising:

• a processor;
• and
• a memory device including instructions that, when executed by the processor, cause the processor to:
• segment information to be indexed into at least one token;
• determine a language type for each token;
• analyze a language dictionary for the determined language type of each token and further segmenting the information into at least one additional token where appropriate based upon contents of the language dictionary;
• search for synonyms in at least one of the multiple languages corresponding to the contents of each token after analyzing the language dictionary for each token;
• and
• store the synonyms and contents of each token for use in indexing the information.

A system for indexing text in multiple language types with synonyms in each language type, comprising:

• a processor;
• and
• a memory device including instructions that, when executed by the processor, cause the processor to:
• receive information to be indexed in at least one language type;
• remove spaces between characters and segmenting the information to be indexed into at least one token;
• determine a language type for each token;
• analyze a language dictionary for the determined language type of each token and further segmenting the information into at least one additional token when contents of the token correspond to separate entries in the language dictionary;
• search for synonyms of the multiple language types corresponding to the contents of each token after analyzing the language dictionary for each token;
• remove punctuation from each token and further segmenting the information into at least one additional token where punctuation is removed;
• and
• store the synonyms and contents of each token for use in indexing the information.

A computer program product embedded in a non-transitory computer readable storage medium and including processor-executable instructions for indexing terms in multiple languages, comprising:

• program code for segmenting information to be indexed into at least one token;
• program code for determining a language type for each token;
• program code for analyzing a language dictionary for the determined language type of each token and further segmenting the information into at least one additional token where appropriate based upon contents of the language dictionary;
• program code for searching for synonyms in at least one of the multiple languages corresponding to the contents of each token after analyzing the language dictionary for each token;
• and
• program code for storing the synonyms and contents of each token for use in indexing the information.