Unsupervised detection of web pages corresponding to a similarity class
AssigneeYahoo! Inc.
InventorMahesh Tiyyagura
Life of this patent
Estimating the term…
Abstract
A method of detecting web pages belonging to at least one similarity class from a plurality of web pages includes determining clusters of the plurality of web pages based on characteristics of the content of the web pages. For each of the determined clusters, at least one metric is determined indicative of similarity among resource locators associated with the web pages of that cluster. A determination of web pages belonging to the at least one similarity class is based on the determined clusters and the determined similarity metrics.
Claims22
A computer implemented method of operating a search system, the method comprising:
- executing a crawl of a plurality of web pages;
- analyzing results of the crawl and detecting web pages of the plurality belonging to at least one similarity class;
- determining clusters of the plurality of web pages based on characteristics of the content of the web pages;
- determining, for each of the determined clusters, at least one metric indicative of similarity among resource locators associated with the web pages of that cluster, wherein the metric indicative of similarity among resource locators is an entropy metric;
- determining the web pages belonging to the at least one similarity class based on the determined clusters and the determined similarity metrics, including determining a cluster having web pages belonging to the at least one similarity class based on local entropy of selected keys of the resource locators for the web pages of that cluster and a statistical measure of the similarity metrics;
- and
- making a notation, based at least in part upon the similarity class, for a web page and corresponding resource locator.
A computer program product for processing search queries and providing search results, the computer program product comprising at least one computer-readable medium having computer program instructions stored therein which are operable to cause at least one computing device to:
- process a plurality of searches and detect web pages belonging to at least one similarity class from a plurality of web pages
- determine clusters of the plurality of web pages based on characteristics of the content of the web pages;
- determine, for each of the determined clusters, at least one metric indicative of similarity among resource locators associated with the web pages of that cluster, wherein the metric indicative of similarity among resource locators is an entropy metric;
- determine the web pages belonging to the at least one similarity class based on the determined clusters and the determined similarity metrics;
- create an indication with a result of the determination;
- and
- fulfill a user initiated search query by making reference to the indication.
Dependencies are inferred from the claim text.