Method and system to extract domain concepts to create domain dictionaries and ontologies

A system for data extraction, comprising:

• one or more hardware processors;
• one or more communication interfaces;
• a relation miner module;
• and
• one or more memory modules storing a plurality of instructions, wherein the plurality of instructions when executed cause the one or more hardware processors to:
• collect a plurality of reference papers from at least one associated source, using at least a plurality of crawlers;
• preprocess a training data to obtain a preprocessed training data, by:
• converting the training data into lower case;
• removing whitespace, punctuations, and commonly occurring words;
• and
• stemming each word in the training data to a root form;
• train a paper area classifier and an intra-paper classifier using the preprocessed training data;
• classify, by the paper area classifier, the plurality of reference papers as relevant or irrelevant based on the preprocessed training data, wherein the paper area classifier analyzes contents in an abstract or a summary of the plurality of reference papers to identify and classify the plurality of reference papers as relevant or irrelevant, and wherein the plurality of reference papers that are classified as irrelevant are discarded;
• for each relevant reference paper:
• identify, by the intra-paper classifier, a plurality of relevant sections in the relevant reference paper based on Frequency-Inverse Document Frequency (Tf-Idf) technique, wherein the intra-paper classifier splits each relevant reference paper into multiple sections and identifies sections within each of the relevant reference papers as a relevant section or an irrelevant section, based on the preprocessed training data;
• pre-process the plurality of relevant sections, via the one or more hardware processors, by performing abbreviation expansion, co-reference resolution, hypernym-hyponym replacements, conversion of complex sentences within the relevant sections into simple sentences, and cleaning of special characters,
• wherein the abbreviation expansion is performed by identifying patterns of occurrences of the abbreviation and their respective expansion pairs,
• wherein the co-reference resolution is performed by identifying and resolving mentions within the plurality of relevant sections using Stanford Corenlp parser,
• wherein the hypernym-hyponym replacements is performed by identifying the hyponyms stated using specific phrases across the plurality of relevant sections, and by replacing a candidate set created for the hypernym-hyponym replacements, and
• wherein the abbreviation expansion and the hypernym-hyponym replacements use lexicon syntactic patterns made up of a combination of keywords and the special characters;
• and
• process the pre-processed plurality of relevant sections, by:
• selecting a plurality of candidate sentences from each pre-processed relevant section;
• identifying a plurality of clauses from each of the plurality of candidate sentences;
• determining at least one relation term in each of the plurality of clauses using the relation miner module, wherein the relation term is at least one of a bidirectional term or a uni-directional term;
• and
• extracting a plurality of qualitative relations and a plurality of quantitative relations from each clause determined as containing the at least one relation term using the relation miner module,
• wherein the plurality of qualitative relations is extracted by identifying a type of a relation statement by considering a position of the relation term in the relation statement, wherein the relation statement indicates a bivariate or a multivariate relation defined between variables,
• wherein the plurality of quantitative relations is extracted by determining strength, confidence, and mean values of the relation statement using a syntactic dependency parsing and the lexicon syntactic patterns, and
• wherein the extracted plurality of qualitative relations and the plurality of quantitative relations are used to create domain dictionaries and ontologies.

A non-transitory computer readable medium embodying a program executable in a computing device for data extraction, comprising:

• collecting a plurality of reference papers from at least one associated source, using at least a plurality of crawlers, via one or more hardware processors;
• preprocessing a training data to obtain a preprocessed training data, by:
• converting the training data into lower case;
• removing whitespace, punctuations, and commonly occurring words;
• and
• stemming each word in the training data to a root form;
• training a paper area classifier and an intra-paper classifier using the preprocessed training data;
• classifying, by the paper area classifier, the plurality of reference papers as relevant or irrelevant based on the preprocessed training data, via the one or more hardware processors, wherein the paper area classifier analyzes contents in an abstract or a summary of the plurality of reference papers to identify and classify the plurality of reference papers as relevant or irrelevant, and wherein the plurality of reference papers that are classified as irrelevant are discarded;
• for each relevant reference paper:
• identifying, by the intra-paper classifier, a plurality of relevant sections in the relevant reference paper based on Frequency-Inverse Document Frequency (Td-Idf) technique, via the one or more hardware processors, wherein the intra-paper classifier splits each relevant reference paper into multiple sections and identifies sections within each of the relevant papers as a relevant section or an irrelevant section, based on the preprocessed training data;
• pre-processing the plurality of relevant sections, via the one or more hardware processors, by performing abbreviation expansion, co-reference resolution, hypernym-hyponym replacements, conversion of complex sentences within the relevant sections into simple sentences, and cleaning of special characters,
• wherein the abbreviation expansion is performed by identifying patterns of occurrences of the abbreviation and their respective expansion pairs,
• wherein the co-reference resolution is performed by identifying and resolving mentions within the plurality of relevant sections using Stanford Corenlp parser,
• wherein the hypernym-hyponym replacements is performed by identifying the hyponyms stated using specific phrases across the plurality of relevant sections, and by replacing a candidate set created for the hypernym-hyponym replacements, and
• wherein the abbreviation expansion and the hypernym-hyponym replacements use lexicon syntactic patterns made up of a combination of keywords and the special characters;
• and
• processing the plurality of pre-processed relevant sections, via the one or more hardware processors, comprising:
• selecting a plurality of candidate sentences from each relevant section;
• identifying a plurality of clauses from each of the plurality of candidate sentences;
• determining at least one relation term in each of the plurality of clauses, wherein the relation term is at least one of a bidirectional term or a uni-directional term;
• and
• extracting a plurality of qualitative relations and a plurality of quantitative relations from each clause determined as containing the at least one relation term,
• wherein the plurality of qualitative relations is extracted by identifying a type of a relation statement by considering a position of the relation term in the relation statement, wherein the relation statement indicates a bivariate or a multivariate relation defined between variables,
• wherein the plurality of quantitative relations is extracted by determining strength, confidence, and mean values of the relation statement using a syntactic dependency parsing and the lexicon syntactic patterns, and
• wherein the extracted plurality of qualitative relations and the plurality of quantitative relations are used to create domain dictionaries and ontologies.