Automated website data collection method
AssigneeNational Taiwan Normal University
InventorsKuo-En CHANG, Yu-Chin LI, Tsung-Chih HU
Life of this patent
Estimating the term…
Abstract
An automated website data collection method uses a hybrid web crawler strategy to obtain a probability distribution of a webpage tag of a webpage of a website to obtains an important feature of the website, and then extracts a text content of important features of the website, and forms a seed vocabulary data set using a composite semantic model. A thematic vocabulary data set having high frequency and highly representative hierarchical structure is further generated by the seed vocabulary data set, and the thematic vocabulary data can be further presented by the visualized system to show the hierarchical structure of thematic vocabulary data set.
Claims7
An automated website data collection method for using an electronic device to crawl a website using a hybrid web crawler to generate a text data set, comprising:
- specifying one of web pages of the website as an analysis web page and obtaining all specified features of the analysis web page;
- selecting a plurality of network addresses associated with the specified features as a web crawling seed node;
- crawling at least one level of the network addresses associated with each web crawling seed node of the website, and selecting a part of the network addresses from the website as a set of associated network addresses;
- selecting a crawling target network address from the set of associated network addresses of the website;
- extracting all webpage tags and corresponding text content associated with the crawling target network address of the website;
- and
- generating the text data set by using the webpage tags and corresponding text content according to a hierarchical structure of the crawling target network address.
Dependencies are inferred from the claim text.