Self-learning based crawling and rule-based data mining for automatic information extraction
AssigneeTata Consultancy Services Limited
InventorsArun Kumar A V, Hemant Kumar Rath, Shameemraj M Nadaf, Anantha Simha
Life of this patent
Estimating the term…
Abstract
Methods and Systems for automatic information extraction by performing self-learning crawling and rule-based data mining is provided. The method determines existence of crawl policy within input information and performs at least one of front-end crawling, assisted crawling and recursive crawling. Downloaded data set is pre-processed to remove noisy data and subjected to classification rules and decision tree based data mining to extract meaningful information. Performing crawling techniques leads to smaller relevant datasets pertaining to a specific domain from multi-dimensional datasets available in online and offline sources.
Claims22
A computer implemented method for automatic information extraction comprising:
- receiving a request for information extraction and retrieving input information from the request;
- determining existence of a crawl policy wherein such determination is performed on the input information retrieved from the request;
- performing assisted crawling, for the input information containing the crawl policy and performing recursive crawling for the input information not containing the crawl policy;
- computing valid paths and links for building a new crawl policy based on the assisted crawling and the recursive crawling, wherein the valid paths and links are computed recursively such that the links in a destination file and a web-page in a current crawling cycle matches with one or more previous crawling attempts;
- pre-processing dataset containing the new crawl policy obtained after the assisted crawling and the recursive crawling to remove noisy data and to obtain a pre-processed dataset;
- and
- subjecting the pre-processed relevant dataset to classification rules and decision tree based data mining to obtain extracted information.
The method of claim-1, wherein the request for automatic information extraction can be system based or user-based.
The method of claim-1, wherein the input information includes product information as at least one of search template and pattern template.
The method of claim-3, wherein the search template includes product names for which information is to be extracted.
The method of claim-3, wherein the pattern template includes patterns leading to destination web-pages.
The method of claim-1, wherein the assisted crawling is performed based on prioritization policy.
The method of claim-1, further comprising:
- determining whether the valid paths and links computed during recursive crawling need to be saved for future attempts of assisted crawling.
The method of claim-1, wherein a complexity of assisted crawling is determined according to an expression:
- Complexity:
- O(n×m+n×k), where
- O is a notation of complexity, n is a number of products to update,
- k is a number of web-pages downloaded and m is a complexity involved for crawling.
The method of claim-1, wherein a complexity of assisted crawling is a function of a number of products to update, a number of web-pages downloaded and a complexity involved for crawling.
The method of claim-1, wherein a complexity of recursive crawling is determined according to an expression:
- Complexity:
- O(n×v2+n×k), where
- O is a notation of complexity, n is a number of products to update,
- k is a number of web-pages downloaded and v is a number of hops required to reach the destination source for recursive crawling.
The method of claim-1, wherein complexity of recursive crawling is a function of the number of products to update, the number of web-pages downloaded and the number of hops required to reach the destination source for recursive crawling.
The method of claim-1, wherein the classification rules includes generic rules and specific rules.
The method of claim-1, wherein new rules can be formulated and added to classification rules.
The method of claim-1, further comprising intimating non-existence of the crawl policy.
The method of claim-1, further comprising:
- intimating any conflict caused due to one or more classification rules;
- and
- extracting information irrespective of type of file formats.
The method of claim-1, further comprising:
- receiving a request for information retrieval and retrieving input information from the request;
- providing the input information for performing front-end crawling;
- pre-processing dataset obtained after the front-end crawling to remove noisy data and to obtain the pre-processed dataset;
- and
- subjecting the pre-processed relevant dataset to classification rules and decision tree based data mining to extract information.
The method of claim-16, wherein the input information includes configuration files containing data dictionary for mapping data source.
The method of claim-16, wherein a complexity of front-end crawling is determined according to an expression:
- Complexity:
- O(n×m+n×k), where
- O is a notation of complexity, n is a number of products to update,
- k is a number of web-pages downloaded and m is a complexity involved for front-end crawling through website or a number of hops to arrive at destination source.
The method of claim-16, wherein a complexity of front-end crawling is a function of a number of products to update, a number of web-pages downloaded and a complexity involved for front-end crawling through website or a number of hops to arrive at destination source.
A computer implemented system for automatic information extraction comprising:
- an input module for receiving a request for information extraction and retrieving information from the request;
- a data source for information extraction;
- one or more processor configured to:
- responsive to the request for information extraction:
- determine the existence of a crawl policy, wherein such determination is performed within input information retrieved from the request;
- perform at least one of the front-end crawling, assisted crawling and recursive crawling on the input information;
- computing valid paths and links for building a new crawl policy based on the assisted crawling and the recursive crawling, wherein the valid paths and links are computed recursively such that the links in a destination file and a web-page in a current crawling cycle matches with one or more previous crawling attempts;
- pre-processing dataset containing the new crawl policy obtained after the assisted crawling and the recursive crawling to remove noisy data and to obtain a pre-processed dataset;
- an extractor to subject pre-processed data to classification rules and decision tree based data mining techniques;
- and
- an output module to provide extracted information.
The system of claim-20, wherein data source include online and offline sources.
A non-transitory computer readable medium embodying a program executable in a computing device for automatic information extraction, the program comprising:
- receiving a request for information extraction and retrieving input information from the request;
- determining existence of a crawl policy wherein such determination is performed on the input information retrieved from the request;
- performing assisted crawling, for the input information containing the crawl policy and performing recursive crawling for the input information not containing the crawl policy;
- computing valid paths and links for building a new crawl policy based on the assisted crawling and the recursive crawling, wherein the valid paths and links are computed recursively such that the links in a destination file and a web-page in a current crawling cycle matches with one or more previous crawling attempts;
- pre-processing dataset containing the new crawl policy obtained after the assisted crawling and the recursive crawling to remove noisy data and to obtain a pre-processed dataset;
- and
- subjecting the pre-processed relevant dataset to classification rules and decision tree based data mining to obtain extracted information.
Dependencies are inferred from the claim text.