US 8,275,790 B2 · System and method of accessing a document efficiently through multi-tier web caching

Abstract

Upon receipt of a document request, a client assistant examines its cache for the document. If not successful, a server searches for the requested document in its cache. If the server copy is still not fresh or not found, the server seeks the document from its host. If the host cannot provide the copy, the server seeks it from a document repository. Certain documents are identified from the document repository as being fresh or stable. Information about each these identified documents is transmitted to the server which inserts entries into an index if the index does not already contain an entry for the document. If and when this particular document is requested, the document will not be present in the server, however the server will contain an entry directing the server to obtain the document from the document repository rather than the document's web host.

Claims42

Dependent claims

A method of accessing a document, comprising:

receiving at a document server a request from a client, the request identifying a URL of a document;
identifying at the document server a first document copy corresponding to the URL, and determining whether the first document copy is stale;
when the first document copy at the document server is determined to be stale, determining at the document server whether a first condition is satisfied using document freshness information stored at the document server;
the first condition including a freshness condition with respect to a repository copy of the document in a search engine repository;
and
when the first document copy at the document server is determined to be stale and the first condition is satisfied,
retrieving the repository copy of the document from the search engine repository;
and
when the first document copy at the document server is determined to be stale, and the first condition is not satisfied,
retrieving a host copy of the document;
wherein the search engine repository is at a location that is independent of the URL and distinct from the document server;
and wherein the search engine repository contains documents, including the repository copy of the document, obtained by a network crawler system.

9 more elements

A method of accessing a document, comprising:

receiving at a document server a request from a client, the request identifying a URL of a document;
identifying at the document server a first document copy corresponding to the URL, and determining whether the first document copy is stale;
when the first document copy at the document server is determined to be stale, and a first condition is satisfied, the first condition including a freshness condition with respect to a repository copy of the document in a search engine repository and a stability condition with respect to the repository copy of the document, wherein the stability condition includes a condition that content of the document has remained unchanged over multiple downloads by a network crawler system,
retrieving the repository copy of the document from the search engine repository;
and
when the first document copy at the document server is determined to be stale, and the first condition is not satisfied,
retrieving a host copy of the document;
wherein
the search engine repository is at a location that is independent of the URL and distinct from the document server;
and wherein the search engine repository contains documents, including the repository copy of the document, obtained by the network crawler system;
satisfying the first condition includes satisfying both the freshness condition and the stability condition;
and
determining whether the document is stale and the first condition is satisfied is performed at the document server.

10 more elements

A system for accessing a document comprising at least one computer, the at least one computer including:

a receiving unit that receives at a document server a request from a client, the request identifying a URL of a document;
a lookup module that identifies at the document server a first document copy corresponding to the URL, and determines whether the first document copy is stale;
instructions for execution by the document server that determine whether a first condition is satisfied using document freshness information stored at the document server when the first document copy at the document server is determined to be stale, wherein the first condition includes a freshness condition with respect to a repository copy of the document in a search engine repository;
and
a document retrieving unit that (A) retrieves the repository copy of the document from the search engine repository when the first document copy is determined to be stale and the first condition applies, and (B) retrieves a host copy of the document when the first document copy is determined to be stale and the first condition is not satisfied;
wherein the search engine repository is at a location that is independent of the URL and distinct from the document server;
and wherein the search engine repository contains documents, including the repository copy of the document, obtained by a network crawler system.

4 more elements

A system for accessing a document comprising at least one computer, the at least one computer including:

a receiving unit that receives at a document server a request from a client, the request identifying a URL of a document;
a lookup module that identifies at the document server a first document copy corresponding to the URL, and determines whether the first document copy is stale;
and
a document retrieving unit that (A) retrieves a repository copy of the document from a search engine repository when the first document copy is determined to be stale and a first condition is satisfied, a first condition including a freshness condition with respect to the repository copy of the document in the search engine repository and a stability condition with respect to the repository copy of the document, wherein the stability condition includes a condition that content of the document has remained unchanged over multiple downloads by the network crawler system, and (B) retrieves a host copy of the document when the first document copy is determined to be stale and the first condition is not satisfied;
wherein
the search engine repository is at a location that is independent of the URL and distinct from the document server;
and wherein the search engine repository contains documents, including the repository copy of the document, obtained by a network crawler system;
satisfying the first condition includes satisfying both the freshness condition and the stability condition;
and
determining whether the document is stale and the first condition is satisfied is performed at the document server.

7 more elements

A non-transitory computer readable storage medium storing one or more computer programs, the one or more computer programs comprising:

instructions for receiving at a document server a request from a client, the request identifying a URL of a document;
instructions for identifying at the document server a first document copy corresponding to the URL, and determining whether the first document copy is stale;
instructions for determining, at the document server, whether a first condition is satisfied using document freshness information stored at the document server when the first document copy at the document server is determined to be stale, wherein the first condition includes a freshness condition with respect to a repository copy of the document in a search engine repository;
and
instructions for retrieving the repository copy of the document from the search engine repository when the first document copy at the document server is determined to be stale and the first condition applies;
and
instructions for retrieving a host copy of the document when the first document copy at the document server is determined to be stale and the first condition is not satisfied,
wherein the search engine repository is at a location that is independent of the URL and distinct from the document server;
and wherein the search engine repository contains documents, including the repository copy of the document, obtained by a network crawler system.

6 more elements

A non-transitory computer readable storage medium storing one or more computer programs, the one or more computer programs comprising:

instructions for receiving at a document server a request from a client, the request identifying a URL of a document;
instructions for identifying at the document server a first document copy corresponding to the URL, and determining whether the first document copy is stale;
instructions for retrieving a repository copy of the document from a search engine repository when the first document copy at the document server is determined to be stale and a first condition is satisfied, the first condition including a freshness condition with respect to the repository copy of the document and a stability condition with respect to the repository copy of the document, wherein the stability condition includes a condition that content of the document has remained unchanged over multiple downloads by a network crawler system;
and
instructions for retrieving a host copy of the document when the first document copy at the document server is determined to be stale and the first condition is not satisfied,
wherein
the search engine repository is at a location that is independent of the URL and distinct from the document server;
and wherein the search engine repository contains documents, including the repository copy of the document, obtained by the network crawler system;
satisfying the first condition includes satisfying both the freshness condition and the stability condition;
and
determining whether the document is stale and the first condition is satisfied is performed at the document server.

8 more elements

A server system, comprising:

one or more processors;
memory storing one or more computer programs, the one or more computer programs comprising:
instructions for receiving at the server system a request from a client, the request identifying a URL of a document;
instructions for identifying at the server system a first document copy corresponding to the URL, and determining whether the first document copy is stale;
instructions for determining at the server system whether a first condition is satisfied using document freshness information stored at the server system when the first document copy at the server system is determined to be stale;
the first condition including a freshness condition with respect to a repository copy of the document in a search engine repository;
and
instructions for retrieving the repository copy of the document from the search engine repository when the first document copy at the server system is determined to be stale and a first condition is satisfied;
and
instructions for retrieving a host copy of the document when the first document copy at the server system is determined to be stale and the first condition is not satisfied,
wherein the search engine repository is at a location that is independent of the URL and distinct from the server system;
and wherein the search engine repository contains documents, including the repository copy of the document, obtained by a network crawler system.