Proxy traffic optimization by caching media resources
AssigneeOxylabs, UAB
InventorTadas GEDGAUDAS
Life of this patent
Estimating the term…
Abstract
Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for caching media resources during a scraping operation. Web resources needed by webpage are stored in a cache that is used by multiple browsers that are scraping the webpage. When an unexpired entry for the web resource is present in the cache, a browser retrieves the web resource and cache instead of making a request from the webpage. This offers a technological improvement of reducing the traffic burden on proxy servers needed to forward the scraping requests and responses.
Claims20
A method for caching web resources during a scraping operation, comprising:
- in response to a first request to scrape, the first request comprising a first address of a target webpage on the Internet:
- (a) by a first browser, retrieving a first content located at the first address, wherein the first content specifies a resource located at a second address, the resource needed to assemble the target webpage;
- (b) by the first browser, retrieving the resource from the Internet at the second address;
- and
- (c) storing the retrieved resource in a local file storage;
- and
- in response to a second request from a client device to scrape, the second request comprising the first address of the target webpage on the Internet:
- (d) by a second browser different from the first browser, retrieving a second content located at the first address;
- (e) by the first or the second browser, determining whether the retrieved second content references the resource at the second address based on the retrieved second content comprising a file name associated with the resource;
- (f) querying a local metadata database based on the file name to retrieve metadata associated with the resource, wherein the metadata indicated that the resource exists in the local file storage;
- (g) determining, based on the metadata, whether the resource is stored unexpired in the local file storage;
- (h) when the resource is determined to be stored unexpired in the local file storage, retrieving the resource from the local file storage to avoid needing to retrieve the resource from the Internet to assemble the target webpage;
- and
- (i) transmitting the resource to the client device.
(canceled)
A non-transitory computer-readable storage medium with instructions stored thereon which, when executed by a computer device, causes the computer device to:
- in response to a first request to scrape, the first request comprising a first address of a target webpage on the Internet:
- (a) by a first browser, retrieving a first content located at the first address, wherein the first content specifies a resource located at a second address, the resource needed to assemble the target webpage;
- (b) by the first browser, retrieving the resource from the Internet at the second address;
- and
- (c) storing the retrieved resource in a local file storage;
- and
- in response to a second request from a client device to scrape, the second request comprising the first address of the target webpage:
- (d) by a second browser different from the first browser, retrieving a second content located at the first address;
- (e) determining whether the retrieved second content references the resource at the second address based on the retrieved second content comprising a file name associated with the resource;
- (f) querying a local metadata database based on the file name to retrieve metadata associated with the resource, wherein the metadata indicated that the resource exists in the local file storage;
- (g) determining, based on the metadata, whether the resource is stored unexpired in the local file storage;
- (h) when the resource is determined to be stored unexpired in the local file storage, retrieving the resource from the local file storage to avoid needing to retrieve the resource from the Internet to assemble the target webpage;
- and
- (i) transmitting the resource to the client device.
(canceled)
Dependencies are inferred from the claim text.