Method and system for extraction of key-terms and synonyms for the key-terms

A key-term and synonyms extracting system (100) for extraction of key-terms and synonyms for the key-terms, the system (100) comprising:

• a memory (128) storing instructions;
• one or more Input/Output (I/O) interfaces (126);
• and
• one or more hardware processors (130) coupled to the memory (128) via the one or more I/O interfaces (126), wherein the one or more hardware processors (130) are configured by the instructions to implement:
• a raw text extractor (102) configured to extract a plurality of raw text from a plurality of text documents using a plurality of parsing techniques, wherein the plurality of text documents is obtained from a plurality of sources;
• a contraction resolver (104) configured to resolve contractions, wherein contractions are identified and replaced in the raw text, wherein any word or any phrase in the raw text that has been shortened by dropping one or more letters is replaced with its full form using a contraction resolution tool;
• an acronym resolver (106) configured to resolve acronyms in the raw text to generate resolved raw terms by identifying atleast one acronym by creating a context window and replacing the identified atleast one acronym with corresponding long forms based on regular expression (regex) techniques;
• a pre-processor (108) configured to pre-process the plurality of resolved raw text, wherein the pre-processing comprises atleast one of tokenizing, part of speech (POS) tagging, filtering punctuations, filtering stop-words, filtering any excluded POS tags, lemmatizing, filtering domain specific words;
• a key-term extractor (112) that further comprises:
• a basic key-term module (114) configured for extracting a basic set of key-terms and a frequency score corresponding to each of the basic set of key-terms from the pre-processed text based on a key-term extraction technique;
• a revised key-term module (116) configured for determining a revised set of key-terms from the basic set of key-terms and a relevancy score from the frequency score corresponding to each of the revised set of key-terms using a reconciliation technique based on a domain language model (110), wherein the domain language model (110) comprises an exhaustive vocabulary data to be used during key-term extraction and synonym extraction, wherein the exhaustive vocabulary data is a vector representation of semantic meaning of a term or a word;
• a final key-term module (118) configured to identify a final set of key-terms from the revised set of key-terms based on the relevancy score, wherein the revised set of key-terms are sorted based on the relevancy score to generate the final set of key-terms;
• a synonym module (120) for configured to determine a set of synonyms for the identified final set of key-terms based on the domain language model (110) or a language model (122) and wherein the synonym module (120) for configured to determine a set of synonyms based on the domain language model (110) or a language model, wherein for determining a set of synonyms for the identified final set of key-terms during availability of the domain language model comprising steps of:
• extracting a stemmed version of the key-term using a stemmer tool (302);
• determining at least one antonym for the key-term using a known tool or a known model (304);
• identifying a set of similar terms from the domain language model based on cosine similarity techniques (306);
• and
• comparing and removing the similar terms that are found to be same as the stemmed version and the determined antonym of the key-terms to obtain the set of synonyms (308);
• and
• a display module (124) configured to display the final set of key-terms and the set of synonyms for the identified final set of key-terms.

A non-transitory computer-readable medium having embodied thereon a computer readable program for extraction of key-terms and synonyms for the key-terms wherein the computer readable program, when executed by one or more hardware processors, cause:

• extracting, by one or more hardware processors, a raw text from a plurality of text documents using a plurality of parsing techniques, wherein the plurality of text documents is obtained from a plurality of sources;
• resolving contractions, wherein the contractions are identified and replaced in the raw text to obtain a refined raw text, wherein any word or any phrase in the raw text that has been shortened by dropping one or more letters is replaced with corresponding full form using a contraction resolution tool;
• resolving acronyms in the refined raw text to generate resolved raw terms by identifying atleast one acronym by creating a context window and replacing the identified atleast one acronym with corresponding long forms based on regular expression (regex) techniques;
• pre-processing the plurality of resolved raw text to obtain a pre-processed text, wherein the pre-processing comprises atleast one of tokenizing, part of speech (POS) tagging, filtering punctuations, filtering stop-words, filtering any excluded POS tags, lemmatizing, filtering domain specific words;
• extracting a basic set of key-terms and a frequency score, corresponding to each of the basic set of key-terms, from the pre-processed text based on a key-term extraction technique, wherein the extracted basic set of key-terms includes a plurality of keywords and a plurality of key-phrases comprising of a plurality of an n-gram terms;
• determining a revised set of key-terms from the basic set of key-terms and a relevancy score from the frequency score, corresponding to each of the revised set of key-terms, using a reconciliation technique based on a domain language model;
• identifying a final set of key-terms from the revised set of key-terms based on the relevancy score, wherein the revised set of key-terms are sorted based on the relevancy score to generate the final set of key-terms;
• determining a set of synonyms for the identified final set of key-terms based on the domain language model or a language model corresponding to the identified final set of key-terms, wherein for determining a set of synonyms for the identified final set of key-terms during availability of the domain language model comprising steps of:
• extracting a stemmed version of the key-term using a stemmer tool;
• determining atleast one antonym for the key-term using a known tool or a known model;
• identifying a set of similar terms from the domain language model based on cosine similarity techniques;
• and
• comparing and removing the similar terms that are found to be same as the stemmed version and the determined antonym of the key-terms to obtain the set of synonyms;
• and
• displaying the final set of key-terms and the set of synonyms for the identified final set of key-terms.