# US 2025/0148032 A1 · System and method for url fetching retry mechanism

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148032 A1
- Document ID: US20250148032A1
- Status: Published application
- Kind: A1
- Application number: 19/012,986
- Filed: 2025-01-08
- Published: 2025-05-08
- Earliest claimed: 2019-02-25
- Assignees: Bright Data Ltd.
- Applicants: Bright Data Ltd.
- Inventors: Derry Shribman; Ofer Vilenski
- Web page: https://patentgrep.com/patent/US20250148032A1

## CPC

- G06F 16/955: using information identifiers, e.g. uniform resource locators [URL]
- G06F 9/547: {Remote procedure calls [RPC]; Web services}
- G06F 16/909: using geographical or spatial information, e.g. location (spatiotemporally dependent retrieval from the web G06F16/9537)
- G06F 16/9558: {Details of hyperlinks; Management of linked annotations}
- G06F 16/9566: {URL specific, e.g. using aliases, detecting broken or misspelled links}
- H04L 1/00: Arrangements for detecting or preventing errors in the information received {(correcting synchronisation H04L7/00)}
- H04L 63/1466: {Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks}
- H04L 67/01: Protocols
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/1014: based on the content of a request
- H04L 67/1021: based on client or server locations
- H04L 67/1036: Load balancing of requests to servers for services different from user content provisioning, e.g. load balancing across domain name servers
- H04L 67/146: Markers for unambiguous identification of a particular session, e.g. session cookie or URL-encoding
- H04L 67/563: Data redirection of data network streams
- H04L 67/564: Enhancement of application control based on intercepted application data
- H04L 67/568: Storing data temporarily at an intermediate stage, e.g. caching
- H04L 69/40: for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection (management of faults, events, alarms or notifications in data switching networks H04L41/06)
- H04L 61/59: using proxies for addressing
- H04L 63/0272: {Virtual private networks}
- H04L 63/145: {the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms}
- H04L 63/1483: {service impersonation, e.g. phishing, pharming or web spoofing (detection of rogue wireless access points H04W12/12)}
- H04L 63/168: {above the transport layer}
- H04L 67/56: Provisioning of proxy services (store-and-forward switching systems in data switching networks H04L12/54)
- H04L 67/63: Routing a service request depending on the request content or context
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A method for overcoming intermittent, temporary, or other fetching failures by using multiple attempts for retrieving a content from a web server to a client device is disclosed. The URL fetching may use direct or non-direct fetching schemes, or a combination thereof. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. Upon sensing a failure of a fetching action, the action is repeated using the same or different parameters or attributes, such as by using different intermediate devices, selected based on different parameters or attributes, such as different countries. The repetitions are limited to a pre-defined maximum number or attempts. The fetching attempts may be performed by the client device, by an intermediate device in a non-direct fetching scheme, or a combination thereof. Various fetching schemes may be used sequentially until the content is retrieved.

## Claims

### Claim 1

A method for use with a content that comprises a web-page or a part thereof, that is stored in a web server, and that is identified by a Uniform Resource Locator (URL), the method comprising:
identifying, by a first device, the URL;
sending, by the first device to the web server via a selected intermediate device over the Internet, a first Hypertext Transfer Protocol (HTTP) request for the content, that comprises the URL, in response to the identifying of the URL, so that the web server is unaware of the first device as the content requesting device;
receiving, by the first device from the web server via a selected intermediate device over the Internet, a first response, in response to the sending of the first HTTP request;
checking a HTTP status code associated with the first response for determining whether the received first response is a proper response, in response to the receiving of the first response; and
responsive to determining that the received first response is not a proper response, performing the steps of:
sending, by the first device to the web server over the Internet using as a source address a selected Internet Protocol (IP) address that is not an IP address of the first device, a second Hypertext Transfer Protocol (HTTP) request for the content, that comprises the URL, so that the web server is unaware of the IP address of the first device as the content requesting device;
receiving, by the first device from the web server over the Internet, a second response, in response to the sending of the second HTTP request; and
checking a HTTP status code associated with the second response for determining whether the received second response is a proper response, in response to the receiving of the second response,
wherein the respective response is determined as a proper response responsive to a HTTP status code of 2xx.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising responsive to determining that the received first or second response is a proper response, sending the respective received first or second response over the Internet.

### Claim 3 (depends on 1)

The method according to claim 1, further comprising responsive to determining that the received first or second response is a proper response, using, by the first device, the respective received first or second response.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the content comprises, or consists of, a part or whole of audio data, voice data, multimedia data, video data, or a computer program.

### Claim 5 (depends on 1)

The method according to claim 1, for use with a group of intermediate devices that includes the selected intermediate device, the method further comprising selecting the selected intermediate device from the group, wherein the sending of the first HTTP request to the web server via the selected intermediate device is in response to the selecting of the selected intermediate device.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the selecting is by the first device.

### Claim 7 (depends on 5)

The method according to claim 5, for use with a list of IP addresses of the intermediate devices in the group, wherein the selecting comprises selecting an IP address of the selected intermediate device from the list.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the list is stored in a first server.

### Claim 9 (depends on 8)

The method according to claim 8, further comprising receiving, from the first server over the Internet, the list.

### Claim 10 (depends on 8)

The method according to claim 8, wherein the identifying comprises receiving of the URL from the first server over the Internet.

### Claim 11 (depends on 8)

The method according to claim 8, wherein the identifying comprises receiving of the URL over the Internet from a device that is different from the first server.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the identifying of the URL comprises receiving of the URL over the Internet.

### Claim 13 (depends on 1)

The method according to claim 1, wherein the first device consists of, includes, is part of, or is integrated with, a proxy server.

### Claim 14 (depends on 13)

The method according to claim 13, wherein the proxy server consists of, includes, is part of, or is integrated with, an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a SOCKS proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof.

### Claim 15 (depends on 1)

The method according to claim 1, wherein the sending of the second HTTP request comprises sending to a proxy server.

### Claim 16 (depends on 15)

The method according to claim 15, wherein the proxy server consists of, includes, is part of, or is integrated with, an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a SOCKS proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof.

### Claim 17 (depends on 1)

The method according to claim 1, wherein the sending of the first HTTP request comprises sending, by the first device to the selected intermediate device over the Internet, the first HTTP request, and wherein the receiving of the first response comprises receiving, by the first device from the selected intermediate device over the Internet, the first response.

### Claim 18 (depends on 1)

The method according to claim 1, for use with a group of intermediate devices that includes the selected intermediate device, and for use with a list of IP addresses that includes the selected IP address, the method further comprising selecting the selected intermediate device from the group, wherein the sending of the first HTTP request to the web server via the selected intermediate device is in response to the selecting of the selected intermediate device, and selecting an IP address from the list, wherein the sending of the second HTTP request using the selected Internet Protocol (IP) address is in response to the selecting of the selected IP address.

### Claim 19 (depends on 18)

The method according to claim 18, wherein the selecting of the selected intermediate device, or the selecting of the selected IP address, is based on, or uses, load balancing.

### Claim 20 (depends on 18)

The method according to claim 18, wherein the selecting of the selected intermediate device, or the selecting of the selected IP address, is based on, or uses, random selection.

### Claim 21 (depends on 20)

The method according to claim 20, wherein random selection uses, or is based on, one or more random numbers generated by a random number generator.

### Claim 22 (depends on 21)

The method according to claim 21, wherein the random number generator is hardware based.

### Claim 23 (depends on 22)

The method according to claim 22, wherein the random number generator is using thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena.

### Claim 24 (depends on 21)

The method according to claim 21, wherein the random number generator is software based.

### Claim 25 (depends on 24)

The method according to claim 24, wherein the random number generator is based on executing an algorithm for generating pseudo-random numbers.

### Claim 26 (depends on 18)

The method according to claim 18, wherein the selecting of the selected intermediate device, or the selecting of the selected IP address, is based on, or uses, an estimated geographical location of a respective selected entity or of the web server.

### Claim 27 (depends on 26)

The method according to claim 26, further comprising estimating a geographical location of the first device, or of the web server, using a geolocation.

### Claim 28 (depends on 27)

The method according to claim 27, wherein the geolocation is based on IP geolocation.

### Claim 29 (depends on 21)

The method according to claim 21, wherein the geolocation is based on W3C Geolocation Application Programming Interface (API).

### Claim 30 (depends on 27)

The method according to claim 27, wherein the selecting is based on estimated as being in the same area as the first device or the web server.

### Claim 31 (depends on 30)

The method according to claim 30, wherein the selecting is based on estimated as being in a same continent, country, state, region, city, postal/zip code, latitude, longitude, or Timezone as the first device or the web server.

### Claim 32 (depends on 18)

The method according to claim 18, wherein the selecting of the selected intermediate device, or the selecting of the selected IP address, is based on being a recent one to be selected, or based on being a least recent to be selected.

### Claim 33 (depends on 1)

The method according to claim 1, wherein the selecting of the selected intermediate device, or the selecting of the selected IP address, is based on manually selecting by a user.

### Claim 34 (depends on 33)

The method according to claim 33, further comprising displaying to the user of at least part of the addresses in a list or devices in the group, and selecting, by the user.

### Claim 35 (depends on 1)

The method according to claim 1, wherein the first or second HTTP request comprises an HTTP Secure (HTTPS) request.

### Claim 36 (depends on 1)

The method according to claim 1, wherein the respective response is determined as not being a proper response responsive to a status code of 4xx or 5xx.

### Claim 37 (depends on 36)

The method according to claim 36, wherein the respective response is determined as not being a proper response responsive to a status code of HTTP 404 error message.

### Claim 38 (depends on 1)

The method according to claim 1, wherein the checking of the respective response comprises using a timeout mechanism.

### Claim 39 (depends on 38)

The method according to claim 38, wherein the respective response is determined as not being a proper response in response to not receiving a proper response after elapsed defined time period after a sending of the URL.

### Claim 40 (depends on 1)

The method according to claim 1, wherein the respective response is determined as not being a proper response in response to detecting an URL redirection.

### Claim 41 (depends on 40)

The method according to claim 40, wherein the URL redirection is identified by checking that the HTTP status code is 3xx Redirection.

### Claim 42 (depends on 1)

The method according to claim 1, for use with a criterion, wherein the determining comprises if the content received satisfies the criterion.

### Claim 43 (depends on 42)

The method according to claim 42, wherein the criterion relates to a feature, characteristic, or type, of the received response.

### Claim 44 (depends on 42)

The method according to claim 42, wherein the criterion comprises a value, and wherein the respective response is determined as not being a proper response in response to comparing a content feature, characteristic, or type, to the value.

### Claim 45 (depends on 44)

The method according to claim 44, wherein the criterion comprises a value of a size of a file, and wherein the response is determined as not being a proper response in response to comparing the received response size to the value.

### Claim 46 (depends on 1)

The method according to claim 1, wherein a communication over the Internet by, or with, the first device or with the selected intermediate device, is based on, uses, or is compatible with, Socket Secure (SOCKS) protocol or connection.

### Claim 47 (depends on 46)

The method according to claim 46, wherein the SOCKS protocol or connection is according to, based on, or is compatible with, SOCKS4, SOCKS4a, or SOCKS5, or wherein the SOCKS protocol or connection is according to, based on, or is compatible with, IETF RFC 1928, IETF RFC 1929, IETF RFC 1961, or IETF RFC 3089.

### Claim 48 (depends on 1)

The method according to claim 1, wherein a communication over the Internet by, or with, the first device or the selected intermediate device, is based on, uses, or is compatible with, WebSocket (ws) or WebSocket Secure (wss) protocol or connection.

### Claim 49 (depends on 1)

The method according to claim 1, wherein the first device or the selected intermediate device is housed in a single enclosure that is a hand-held enclosure or a portable enclosure.

### Claim 50 (depends on 1)

The method according to claim 1, wherein the first device or the selected intermediate device consists of, comprises, is part of, or is integrated with, a notebook computer, a laptop computer, a media player, a Digital Still Camera (DSC), a Digital video Camera (DVC or digital camcorder), a Personal Digital Assistant (PDA), a cellular telephone, a digital camera, a video recorder, or a smartphone.

### Claim 51 (depends on 50)

The method according to claim 50, wherein the first device or the selected intermediate device consists of, comprises, is part of, or is integrated with, a smartphone that comprises, or is based on, an Apple iPhone 6 or a Samsung Galaxy S6.

### Claim 52 (depends on 1)

The method according to claim 1, further comprising storing, operating, or using, by the first device, an operating system.

### Claim 53 (depends on 52)

The method according to claim 52, wherein the operating system comprises a mobile operating system.

### Claim 54 (depends on 53)

The method according to claim 53, wherein the mobile operating system is based on, or comprises, Android version 2.2 (Froyo), Android version 2.3 (Gingerbread), Android version 4.0 (Ice Cream Sandwich), Android Version 4.2 (Jelly Bean), Android version 4.4 (KitKat), Apple iOS version 3, Apple iOS version 4, Apple ios version 5, Apple iOS version 6, Apple iOS version 7, Microsoft Windows® Phone version 7, Microsoft Windows® Phone version 8, Microsoft Windows® Phone version 9, or Blackberry® operating system.

### Claim 55 (depends on 1)

The method according to claim 1, further comprising storing, operating, or using, by the first device or by the selected intermediate device, a web browser.

### Claim 56 (depends on 55)

The method according to claim 55, wherein the web browser consists of, comprises of, or is based on, Microsoft Internet Explorer, Google Chrome, Opera™, or Mozilla Firefox®.

### Claim 57 (depends on 55)

The method according to claim 55, wherein the web browser is a mobile web browser.

### Claim 58 (depends on 57)

The method according to claim 57, wherein the mobile web browser consists of, comprises of, or is based on, Safari, Opera Mini™, or Android web browser.

### Claim 59 (depends on 55)

The method according to claim 55, wherein further comprising returning an error message to the web browser.

### Claim 60 (depends on 1)

The method according to claim 1, wherein the identifying comprises identifying the URL by a web browser.

### Claim 61 (depends on 60)

The method according to claim 60, wherein the identifying is performed as part of a plug-in or an extension integrated with the web browser.

### Claim 62 (depends on 61)

The method according to claim 61, wherein at least one of the steps is performed by the web browser that is integrated in a form of a plug-in or an extension.

### Claim 63 (depends on 61)

The method according to claim 61, wherein the identifying of the URL uses a plug-in or an extension to the web browser.

### Claim 64 (depends on 61)

The method according to claim 61, wherein the integration is by hooking to the web browser, wherein the integration is in a filter driver form, or, wherein the web browser and the steps are communicating using an Inter-Process Communication (IPC).

### Claim 65 (depends on 64)

The method according to claim 64, wherein the identifying of the URL uses hooking to the web browser, a filter driver form, or using an Inter-Process Communication (IPC).

### Claim 66 (depends on 65)

The method according to claim 65, wherein the IPC is using a file sharing, a signal, a socket, a pipe, a message queue, a shared memory, a semaphore, or memory mapped file.

### Claim 67 (depends on 65)

The method according to claim 65, wherein the IPC is using a clipboard, a Component Object Model (COM), a data copy, a DDE protocol, or mailslots.

### Claim 68 (depends on 1)

The method according to claim 1, wherein the first device or the selected intermediate device is integrated in part or entirely within an appliance.

### Claim 69 (depends on 1)

The method according to claim 1, wherein the first device or the selected intermediate device, consists of, comprises, is integrated with, or is part of, a wearable device that is wearable on a person.

### Claim 70 (depends on 1)

The method according to claim 1, further comprising storing, operating, or using, by the first device or by the selected intermediate device, a client operating system.

### Claim 71 (depends on 70)

The method according to claim 70, wherein the client operating system consists of, comprises, or is based on, one out of Microsoft Windows 7, Microsoft Windows XP, Microsoft Windows 8, Microsoft Windows 8.1, Linux, and Google Chrome OS.

### Claim 72 (depends on 70)

The method according to claim 70, wherein the client operating system is a Real-Time Operating System (RTOS).

### Claim 73 (depends on 72)

The method according to claim 72, wherein the RTOS comprises FreeRTOS, SafeRTOS, QNX, VxWorks, or Micro-Controller Operating Systems (μC/OS).
