Path-based ranking of unvisited web pages
AssigneeIBM Corporation
InventorsXiaochuan Ma, Yue Pan, Hui Su
Life of this patent
Estimating the term…
Abstract
Path-based ranking of unvisited Web pages for WWW crawling is provided, via identifying all the paths beginning with a “seed” URL and leading to visited relevant web pages as “good-path set”, and for each unvisited web page, identifying the paths beginning from the “seed” URL leading to it as “partial-path set”; classifying all the visited web pages and labeling each web Page with the labels of a class or classes it belongs to; training a statistic model for generalizing the common patterns among all ones of “good-path set”; and evaluating the “partial-path set” with the statistic model and ranking the unvisited web pages with the evaluation results.
Claims9
A method for path-based ranking of unvisited Web pages for WWW crawling, comprising:
- identifying all the paths beginning with a “seed” URL and leading to visited relevant web pages as “good-path set”, and for each unvisited web page, identifying the paths beginning from the “seed” URL leading to it as “partial-path set”;
- classifying all the visited web pages and labeling each web page with the labels of a class or classes it belongs to;
- training a statistic model for generalizing the common patterns among all ones of good-path set;
- and
- evaluating the partial-path sets with the statistic model and ranking the unvisited web pages with the evaluation results.
A method for deploying computing infrastructure, comprising integrating computer readable code into a computing system, wherein the code in combination with the computing system is capable of performing a method for path-based ranking of unvisited Web pages for WWW crawling, comprising at least one of the following:
- identifying all the paths beginning with a “seed” URL and leading to visited relevant web pages as “good-path set”, and for each unvisited web page, identifying the paths beginning from the “seed” URL leading to it as “partial-path set”;
- classifying all the visited web pages and labeling each web page with the labels of a class or classes it belongs to;
- training a statistic model for generalizing the common patterns among all ones of good-path set;
- and
- evaluating the partial-path sets with the statistic model and ranking the unvisited web pages with the evaluation results.
A system for path-based ranking of unvisited Web pages for WWW crawler, comprising:
- a storer which stores “good-path set” and “partial-path set”;
- a classifier and a labeler which classifies all the visited web pages and labels each web page with the labels of a class or classes it belongs to;
- a statistic modeler which trains a statistic model for generalizing the common patterns among all ones of “good-path set”;
- and
- an evaluator which evaluates the “partial-path set” with the statistic model and ranking the unvisited web pages with the evaluation results.
A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for path-based ranking of unvisited Web pages for WWW crawling, said method comprising the steps of:
- identifying all the paths beginning with a “seed” URL and leading to visited relevant web pages as “good-path set”, and for each unvisited web page, identifying the paths beginning from the “seed” URL leading to it as “partial-path set”;
- classifying all the visited web pages and labeling each web page with the labels of a class or classes it belongs to;
- training a statistic model for generalizing the common patterns among all ones of good-path set;
- and
- evaluating the partial-path sets with the statistic model and ranking the unvisited web pages with the evaluation results.
Dependencies are inferred from the claim text.