Identifying equivalent links on a page
AssigneeInternational Business Machines Corporation
InventorsKhalil A. Ayoub, Paul Ionescu, Iosif Viorel Onut, Wayne Duncan Smith
Life of this patent
Estimating the term…
Abstract
A computer-implemented process for identifying equivalent links on a page responsive to a determination that the crawler has not visited all required universal resource locators, locates a next URL to be crawled to form a current URL and processes the current URL to identify equivalent URLs. Responsive to a determination that the crawler has not visited the current URL, determine whether necessary to crawl all identified equivalent URLs and responsive to a determination that it is necessary to crawl all identified equivalent URLs, adding all equivalent URLs to a list of URLs to be crawled.
Claims20
A computer-implemented process for identifying equivalent links on a page, the computer-implemented process comprising:
- determining whether a crawler visited all required universal resource locators;
- responsive to a determination that the crawler has not visited all required universal resource locators, locating a next URL to be crawled to form a current URL;
- processing the current URL, using a processor, to identify equivalent URLs;
- determining whether the crawler visited the current URL;
- responsive to a determination that the crawler has not visited the current URL, determining whether necessary to crawl all identified equivalent URLs;
- and
- responsive to a determination that it is necessary to crawl all identified equivalent URLs, adding all identified equivalent URLs to a list of URLs to be crawled.
A computer program product for identifying equivalent links on a page, the computer program product, comprising:
- a computer hardware storage device containing computer executable program code stored thereon, the computer executable program code comprising:
- computer executable program code for determining whether a crawler visited all required universal resource locators;
- computer executable program code responsive to a determination that the crawler has not visited all required universal resource locators, for locating a next URL to be crawled to form a current URL;
- computer executable program code for processing the current URL to identify equivalent URLs;
- computer executable program code for determining whether the crawler visited the current URL;
- computer executable program code responsive to a determination that the crawler has not visited the current URL, for determining whether necessary to crawl all identified equivalent URLs;
- and
- computer executable program code responsive to a determination that it is necessary to crawl all identified equivalent URLs, for adding all identified equivalent URLs to a list of URLs to be crawled.
An apparatus for identifying equivalent links on a page, the apparatus comprising:
- a communications fabric;
- a memory connected to the communications fabric, wherein the memory contains computer executable program code;
- a communications unit connected to the communications fabric;
- an input/output unit connected to the communications fabric;
- and
- a processor unit connected to the communications fabric, wherein the processor unit executes the computer executable program code to direct the apparatus to:
- determine whether a crawler visited all required universal resource locators;
- responsive to a determination that the crawler has not visited all required universal resource locators, locate a next URL to be crawled to form a current URL;
- process the current URL to identify equivalent URLs;
- determine whether the crawler visited the current URL;
- responsive to a determination that the crawler has not visited the current URL, determine whether necessary to crawl all identified equivalent URLs;
- and
- responsive to a determination that it is necessary to crawl all identified equivalent URLs, add all identified equivalent URLs to a list of URLs to be crawled.
Dependencies are inferred from the claim text.