US 6,424,966 B1 · Synchronizing crawler with notification source

Abstract

A method and system for the processing and maintenance of electronic information retrieved from electronic documents stored on a computer network. The gatherer program of the present invention employs a crawler to crawl a portion of the computer network to retrieve electronic documents found during the crawl and that meet a set of crawl restriction rules. Some or all of the data contained in the copies of electronic documents is then stored in a data store such as an index. The invention keeps the data in the data store current by accepting notifications of when a previously retrieved document has changed. The notifications are sent by a notification source that monitors a space containing the previously retrieved documents for changes occurring after the document was last retrieved by the gatherer program. Because the document is being monitored for changes by the notification source, the gatherer program only needs to retrieve the document again when the gatherer program has been notified that the document has changed. If the notification source experiences a discontinuity, such as a system shutdown, the notification source requests that the gatherer perform an initialization crawl to retrieve any documents that changed while the notification source was not operational.

Claims25

Dependent claims

A computer-based method of retrieving and maintaining information above electronic documents stored on a computer network, each electronic document having an associated document address specification, the method comprising:

(a) retrieving information about selected electronic documents, including their document address specification, from the computer network, for each selected document the retrieving including:
(i) adding the associated document address specification to a transaction log;
(ii) returning a copy of the electronic document and marking the document address specification in the transaction log;
(iii) parsing the returned electronic copy to identify links to other electronic documents having document address specifications;
(iv) adding the document address specifications of the other electronic documents to the transaction log unless previously in the transaction log;
and
(v) repeating (ii)-(iv) until there are no unmarked document address specifications in the transaction log;
(b) storing at least some of the data associated with each returned electronic document copy in a data store, the document address specification being associated with the data for retrieval of the data from the data store;
(c) providing a notification source for monitoring the document address specification corresponding to each returned electronic document copy for a change made to the electronic document associated with the document address specification;
(d) sending a notification message from the notification source when the monitoring by a notification source detects a change in the electronic document data associated with the document address specification;
(e) processing the notification message so as to cause an electronic document copy containing the detected changes to be returned;
and
(f) updating at least some of the data associated with the returned electronic copy in the data store based on the returned electronic document copy containing the detected changes.

10 more elements

A computer-based system for retrieving and maintaining information associated with a plurality of electronic documents stored on a computer network, the system comprising:

(a) a gatherer for performing an initialization crawl, said initialization crawl comprising:
(i) adding a document address specification to a transaction log;
(ii) retrieving a copy of a source electronic document from a location on the computer network, the location of the source electronic document defined by a document address specification currently in the transaction log and marking the document address specification in the transaction log;
(iii) parsing the retrieved source electronic document to identify links to other source electronic documents having document address specifications;
(iv) adding the document address specifications of the other source electronic documents to the transaction log unless previously in the transaction log;
(v) storing at least some of an original information content from the copy of the source electronic document in a data store;
(vi) associating the original information content that is stored in the data store with the document address specification of the source electronic document;
and
(vii) repeating (ii)-(vi) until there are no unmarked document address specifications in the transaction log;
(b) a notification source for monitoring the source electronic document stored at a document address specification, said monitoring the source electronic document comprising:
(i) detecting when a change has been made to the original information content of the source electronic document;
and
(ii) sending a notification message to the gatherer when the notification source detects that the source electronic document has been changed, the notification message including the document address specification of the source electronic document that has been changed;
and
(c) said notification retrieval being performed by said gatherer when said gatherer receives a notification message, said notification retrieval comprising:
(i) retrieving a second copy of the source electronic document from a location on said computer networks;
(ii) updating at least some of the original information content in the data store based on the retrieved second copy of the source electronic document;
and
(iii) maintaining the association of the updated original information contained in the data store with the document address specification of the source electronic document.

16 more elements

A computer readable medium having computer-executable instructions for retrieving and maintaining information about electronic documents stored on a computer network, each electronic document having an associated document address specification, comprising:

(a) performing an initialization crawl, the initialization crawl comprising:
(i) adding a document address specification to a transaction log;
(ii) returning a copy of the electronic document associated with the document address specification and marking the document address specification in the transaction log;
(iii) parsing the returned electronic document copy to identify links to other electronic documents having document address specifications;
(iv) adding the document address specifications of the other electronic documents to the transaction log unless previously in the transaction log;
and
(v) repeating (ii)-(iv) until there are not unmarked document address specifications in the transaction log;
(b) monitoring with a notification source the electronic documents associated with said document address specifications for a change to the electronic document;
(c) sending a notification message from the notification source when an electronic document has changed;
and
(d) performing a notification retrieval inn response to the notification message, the notification retrieval comprising retrieving a new copy of the electronic document from the computer network.

8 more elements

A computer-based system for retrieving and maintaining information about electronic documents stored on a computer network, each of said electronic documents having a documents address specification, the system comprising:

(a) means for performing an initialization crawl including:
(i) means for adding a document address specification to a transaction log;
(ii) means for returning a copy of the electronic document associated with the document address specification and marking the document address specification in the transaction log;
(iii) means for parsing the returned electronic document copy to identify links to other electronic documents having document address specifications;
(iv) means for adding the document address specifications of the other electronic documents to the transaction log unless previously in the transaction log;
and
(v) means for repeating (ii)-(iv) until there are no unmarked document address specifications in the transaction log;
(b) means for detecting changes in the electronic documents associated with said document address specifications;
and
(c) means for performing an notification retrieval in response to the detection of a change in an electronic document, the notification retrieval comprising retrieving a new copy of the electronic document from the computer network.

7 more elements

Dependencies are inferred from the claim text.

Synchronizing crawler with notification source

Life of this patent

Synchronizing crawler with notification source

Life of this patent

Abstract

Claims25