US 9,430,567 B2 · Identifying unvisited portions of visited information

Abstract

Identifying unvisited portions of visited information to visit includes receiving information to crawl, wherein the information is representative of one of web based information and non-web based information, computing a locality sensitive hash (LSH) value for the received information, and identifying a most similar information visited thus far. Identifying unvisited portions of visited information further includes determining whether the LSH of the received information is equivalent to most similar information visited thus far and, responsive to a determination that the LSH of the received information is not equivalent to most similar information visited thus far, identifying a visited portion of the received information using information for most similar information visited thus far and crawling only unvisited portions of the received information.

Claims20

Dependent claims

A computer-implemented process for identifying unvisited portions of visited information to visit, the computer-implemented process comprising:

receiving information to crawl, wherein the information is representative of one of web-based information and non-web based information;
computing, using a processor, a locality sensitive hash (LSH) value for the received information;
identifying a most similar information visited thus far;
determining whether the LSH of the received information is equivalent to most similar information visited thus far;
responsive to a determination that the LSH of the received information is not equivalent to most similar information visited thus far, identifying a visited portion of the received information using information for most similar information visited thus far;
and
crawling only unvisited portions of the received information.

4 more elements

A computer program product for identifying unvisited portions of visited information to visit, the computer program product comprising:

a computer recordable-type media containing computer executable program code stored thereon, the computer executable program code comprising:
computer executable program code for receiving information to crawl, wherein the information is representative of one of web-based information and non-web based information;
computer executable program code for computing a locality sensitive hash (LSH) value for the received information;
computer executable program code for identifying a most similar information visited thus far;
computer executable program code for determining whether the LSH of the received information is equivalent to most similar information visited thus far;
computer executable program code responsive to a determination that the LSH of the received information is not equivalent to most similar information visited thus far, identifying a visited portion of the received information using information for most similar information visited thus far;
and
computer executable program code for crawling only unvisited portions of the received information.

5 more elements

An apparatus for identifying unvisited portions of visited information to visit, the apparatus comprising:

a communications fabric;
a memory connected to the communications fabric, wherein the memory contains computer executable program code;
a communications unit connected to the communications fabric;
an input/output unit connected to the communications fabric;
a display connected to the communications fabric;
and
a processor unit connected to the communications fabric, wherein the processor unit executes the computer executable program code to direct the apparatus to:
receive information to crawl, wherein the information is representative of one of web-based information and non-web based information;
compute a locality sensitive hash (LSH) value for the received information;
identify a most similar information visited thus far;
determine whether the LSH of the received information is equivalent to most similar information visited thus far;
responsive to a determination that the LSH of the received information is not equivalent to most similar information visited thus far, identify a visited portion of the received information using information for most similar information visited thus far;
and
crawl only unvisited portions of the received information.

11 more elements

Dependencies are inferred from the claim text.

Identifying unvisited portions of visited information

Life of this patent

Identifying unvisited portions of visited information

Life of this patent

Abstract

Claims20