# US 2025/0148033 A1 · System and method for url fetching retry mechanism

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148033 A1
- Document ID: US20250148033A1
- Status: Published application
- Kind: A1
- Application number: 19/012,991
- Filed: 2025-01-08
- Published: 2025-05-08
- Earliest claimed: 2019-02-25
- Assignees: Bright Data Ltd.
- Applicants: Bright Data Ltd.
- Inventors: Derry Shribman; Ofer Vilenski
- Web page: https://patentgrep.com/patent/US20250148033A1

## CPC

- G06F 16/955: using information identifiers, e.g. uniform resource locators [URL]
- G06F 9/547: {Remote procedure calls [RPC]; Web services}
- G06F 16/909: using geographical or spatial information, e.g. location (spatiotemporally dependent retrieval from the web G06F16/9537)
- G06F 16/9558: {Details of hyperlinks; Management of linked annotations}
- G06F 16/9566: {URL specific, e.g. using aliases, detecting broken or misspelled links}
- H04L 1/00: Arrangements for detecting or preventing errors in the information received {(correcting synchronisation H04L7/00)}
- H04L 63/1466: {Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks}
- H04L 67/01: Protocols
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/1014: based on the content of a request
- H04L 67/1021: based on client or server locations
- H04L 67/1036: Load balancing of requests to servers for services different from user content provisioning, e.g. load balancing across domain name servers
- H04L 67/146: Markers for unambiguous identification of a particular session, e.g. session cookie or URL-encoding
- H04L 67/563: Data redirection of data network streams
- H04L 67/564: Enhancement of application control based on intercepted application data
- H04L 67/568: Storing data temporarily at an intermediate stage, e.g. caching
- H04L 69/40: for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection (management of faults, events, alarms or notifications in data switching networks H04L41/06)
- H04L 61/59: using proxies for addressing
- H04L 63/0272: {Virtual private networks}
- H04L 63/145: {the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms}
- H04L 63/1483: {service impersonation, e.g. phishing, pharming or web spoofing (detection of rogue wireless access points H04W12/12)}
- H04L 63/168: {above the transport layer}
- H04L 67/56: Provisioning of proxy services (store-and-forward switching systems in data switching networks H04L12/54)
- H04L 67/63: Routing a service request depending on the request content or context
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A method for overcoming intermittent, temporary, or other fetching failures by using multiple attempts for retrieving a content from a web server to a client device is disclosed. The URL fetching may use direct or non-direct fetching schemes, or a combination thereof. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. Upon sensing a failure of a fetching action, the action is repeated using the same or different parameters or attributes, such as by using different intermediate devices, selected based on different parameters or attributes, such as different countries. The repetitions are limited to a pre-defined maximum number or attempts. The fetching attempts may be performed by the client device, by an intermediate device in a non-direct fetching scheme, or a combination thereof. Various fetching schemes may be used sequentially until the content is retrieved.

## Claims

### Claim 1

A method comprising:
receiving, by a first device over the Internet, a Uniform Resource Locator (URL) that identifies a content that comprises a web-page or a part thereof, and that is stored only in a web server;
selecting, by the first device, a first Internet protocol (IP) address from a first list of IP addresses;
obtaining, by the first device from the web server over the Internet, in response to a first request for the content that uses the selected first IP address as a source address, so that the IP address of the first device is unknown to the web server as a requesting device, a first response;
determining whether the first response is a proper response that includes a HTTP status code of 2xx; and
responsive to determining that the first response is not a proper response:
obtaining, by the first device from the web server over the Internet, in response to a second request for the content that uses as a source address a second IP address that is different from the first IP address, so that the IP address of the first device is unknown to the web server as a requesting device, a second response;
checking whether the second response is a proper response that includes a HTTP status code of 2xx;
responsive to determining that the second response is a proper response, using the content received in the second response; and
responsive to determining that the second response is not a proper response, performing, by the first device, a first action,
wherein the content comprises a computer file, audio data, voice data, multimedia data, video data, an image, music, or a computer program.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising responsive to determining that the first response is a proper response, using, by the first device, the content received in the first response.

### Claim 3 (depends on 1)

The method according to claim 1, further comprising selecting, by the first device, the second IP address from a second list of IP addresses.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the obtaining of the first response comprises:
sending, by the first device to the web server over the Internet, a first Hypertext Transfer Protocol (HTTP) request that comprises the URL; and
receiving, by the first device over the Internet, the first response in response to the sent first HTTP request.

### Claim 5 (depends on 4)

The method according to claim 4, wherein each of the IP addresses in the first list addresses a respective device, and wherein the obtaining of the first response comprises:
sending, by the first device over the Internet to a second device that is addresses by the selected first IP address, the first HTTP request;
receiving, by the second device over the Internet, the first HTTP request sent by the first device;
sending, by the second device to the web server over the Internet, the first HTTP request;
receiving, by the second device from the web server over the Internet, the first response;
sending, by the second device to the first device over the Internet, the received first response; and
receiving, by the first device from the second device over the Internet, the sent first response.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the obtaining of the second response comprises:
sending, by the first device to the web server over the Internet, a second Hypertext Transfer Protocol (HTTP) request that comprises the URL; and
receiving, by the first device over the Internet, the second response in response to the sent second HTTP request.

### Claim 7 (depends on 6)

The method according to claim 6, the second IP addresses in addresses a second device, and wherein the obtaining of the second response comprises:
sending, by the first device over the Internet to a second device, the second HTTP request;
receiving, by the second device over the Internet, the second HTTP request sent by the first device;
sending, by the second device to the web server over the Internet, the second HTTP request;
receiving, by the second device from the web server over the Internet, the second response;
sending, by the second device to the first device over the Internet, the received second response; and
receiving, by the first device from the second device over the Internet, the sent second response.

### Claim 8 (depends on 1)

The method according to claim 1, further comprising:
obtaining, by the first device from the web server over the Internet, in response to a first request for the content that uses the IP address of the first device as a source address, a third response;
determining whether the third response is a proper response that comprises the content or that includes a status code of 2xx; and
responsive to determining that the third response is a proper response, using, by the first device, the content in the third response.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the first action comprises the obtaining of the third response.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the using of the content comprises sending, by the first device over the Internet, the content.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the receiving of the URL comprises receiving, by the first device from a server device over the Internet, the URL.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the using of the content comprises sending, by the first device to the server device over the Internet, the content.

### Claim 13 (depends on 1)

The method according to claim 1, wherein the receiving of the URL comprises receiving, by the first device from a client device over the Internet, the URL.

### Claim 14 (depends on 13)

The method according to claim 13, wherein the using of the content comprises sending, by the first device to the client device over the Internet, the content.

### Claim 15 (depends on 1)

The method according to claim 1, wherein the first device comprises a proxy server or a Data-Center (DC) proxy-server.

### Claim 16 (depends on 15)

The method according to claim 15, wherein the proxy server consists of, includes, is part of, or is integrated with, an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a SOCKS proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof.

### Claim 17 (depends on 1)

The method according to claim 1, wherein the first device is a client device.

### Claim 18 (depends on 1)

The method according to claim 1, wherein the selecting of the first address is based on, or uses, random selection.

### Claim 19 (depends on 1)

The method according to claim 1, wherein the selecting of the first IP address is based on, or uses, an estimated geographical location of the first device or of the web server.

### Claim 20 (depends on 19)

The method according to claim 19, further comprising estimating a geographical location of the first device or of the web server using geolocation, that is based on IP geolocation that uses W3C Geolocation Application Programming Interface (API).

### Claim 21 (depends on 20)

The method according to claim 20, wherein the selecting of the first IP address is based on estimated as being in a same region as the first device or the web server.

### Claim 22 (depends on 20)

The method according to claim 20, wherein the region comprises a country, state, region, city, postal/zip code, latitude, longitude, or Timezone.

### Claim 23 (depends on 1)

The method according to claim 1, wherein selecting of at the first IP address is based on being a respective recent one to be selected, or based on being the least respective recent to be selected.

### Claim 24 (depends on 1)

The method according to claim 1, wherein the selecting of the first IP address comprises manually selecting by a user.

### Claim 25 (depends on 1)

The method according to claim 1, wherein the respective response is determined as not being a proper response responsive to a status code of 4xx or 5xx.

### Claim 26 (depends on 1)

The method according to claim 1, wherein the respective response is determined as not being a proper response responsive to a status code of HTTP 404 error message.

### Claim 27 (depends on 1)

The method according to claim 1, wherein the determining of the respective response comprises using a timeout mechanism.

### Claim 28 (depends on 27)

The method according to claim 27, wherein the response is determined as not being a proper response in response to not receiving a proper response after elapsed defined time period after the sending of the respective request.

### Claim 29 (depends on 23)

The method according to claim 23, wherein the determining comprises determining if an URL redirection is identified, and wherein the respective response is determined as not being a proper response in response to detecting the URL redirection, and wherein the URL redirection is identified by checking that the HTTP status code is 3xx Redirection.

### Claim 30 (depends on 1)

The method according to claim 1, for use with a criterion, wherein the determining of the respective response comprises verifying if the content received in the respective response satisfies the criterion, and wherein the criterion relates to a feature, characteristic, or type, of the received content in the respective response.

### Claim 31 (depends on 1)

The method according to claim 1, further for use with a virtualization, wherein the first device consists of, comprises, is part of, or is integrated with, a server device that virtualizes a client device that is addressed by the selected first IP address.

### Claim 32 (depends on 31)

The method according to claim 31, wherein the client device virtualization executed as part of a Virtual Machine (VM).

### Claim 33 (depends on 32)

The method according to claim 32, for use with a host computer that implements the VM, wherein the method further comprising executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM).

### Claim 34 (depends on 32)

The method according to claim 32, wherein the virtualization includes, is based on, or uses, full virtualization, para-virtualization, or hardware assisted virtualization.
