# US 2025/0148034 A1 · System and method for url fetching retry mechanism

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148034 A1
- Document ID: US20250148034A1
- Status: Published application
- Kind: A1
- Application number: 19/017,670
- Filed: 2025-01-12
- Published: 2025-05-08
- Earliest claimed: 2019-02-25
- Assignees: Bright Data Ltd.
- Applicants: Bright Data Ltd.
- Inventors: Derry Shribman; Ofer Vilenski
- Web page: https://patentgrep.com/patent/US20250148034A1

## CPC

- G06F 16/955: using information identifiers, e.g. uniform resource locators [URL]
- G06F 9/547: {Remote procedure calls [RPC]; Web services}
- G06F 16/909: using geographical or spatial information, e.g. location (spatiotemporally dependent retrieval from the web G06F16/9537)
- G06F 16/9558: {Details of hyperlinks; Management of linked annotations}
- G06F 16/9566: {URL specific, e.g. using aliases, detecting broken or misspelled links}
- H04L 1/00: Arrangements for detecting or preventing errors in the information received {(correcting synchronisation H04L7/00)}
- H04L 63/1466: {Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks}
- H04L 67/01: Protocols
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/1014: based on the content of a request
- H04L 67/1021: based on client or server locations
- H04L 67/1036: Load balancing of requests to servers for services different from user content provisioning, e.g. load balancing across domain name servers
- H04L 67/146: Markers for unambiguous identification of a particular session, e.g. session cookie or URL-encoding
- H04L 67/563: Data redirection of data network streams
- H04L 67/564: Enhancement of application control based on intercepted application data
- H04L 67/568: Storing data temporarily at an intermediate stage, e.g. caching
- H04L 69/40: for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection (management of faults, events, alarms or notifications in data switching networks H04L41/06)
- H04L 61/59: using proxies for addressing
- H04L 63/0272: {Virtual private networks}
- H04L 63/145: {the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms}
- H04L 63/1483: {service impersonation, e.g. phishing, pharming or web spoofing (detection of rogue wireless access points H04W12/12)}
- H04L 63/168: {above the transport layer}
- H04L 67/56: Provisioning of proxy services (store-and-forward switching systems in data switching networks H04L12/54)
- H04L 67/63: Routing a service request depending on the request content or context
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A method for overcoming intermittent, temporary, or other fetching failures by using multiple attempts for retrieving a content from a web server to a client device is disclosed. The URL fetching may use direct or non-direct fetching schemes, or a combination thereof. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. Upon sensing a failure of a fetching action, the action is repeated using the same or different parameters or attributes, such as by using different intermediate devices, selected based on different parameters or attributes, such as different countries. The repetitions are limited to a pre-defined maximum number or attempts. The fetching attempts may be performed by the client device, by an intermediate device in a non-direct fetching scheme, or a combination thereof. Various fetching schemes may be used sequentially until the content is retrieved.

## Claims

### Claim 1

A method comprising:
identifying, by a first device, a Uniform Resource Locator (URL) of a content that is stored in a web server;
sending, by the first device to a first server over the Internet, a first request for the content that comprises the URL, in response to the identifying of the URL;
receiving, by the first device from the web server via the first server over the Internet, a first response, in response to the sending of the first request;
determining, by the first device, whether the received first response is a proper response that comprises the content, in response to the receiving of the first response;
sending, by the first device to the web server over the Internet, a second request for the content that includes the URL, using an Internet Protocol (IP) address of the first device as a source address, in response to determining that the received first response is not a proper response;
receiving, by the first device from the web server over the Internet, a second response, in response to the sending of the second request; and
determining, by the first device, whether the received second response is a proper response that comprises the content, in response to the receiving of the second response.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
receiving, by the first server from the first device over the Internet, the first request, in response to the sending of the first request;
selecting, by the first server, a first IP address from a list of IP addresses;
sending, by the first server to the web server over the Internet, using the selected first IP address as the source address, a third request for the content that includes the URL;
receiving, by the first server from the web server over the Internet, the first response, in response to the sending of the third request; and
sending, by the first server to the first device over the Internet, the first response received by the first server, in response to the receiving of the first response by the first server,
wherein the receiving by the first device from the web server of the first response is in response to the sending of the first request by the first server.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the sending by the first server to the web server of the third request comprises sending, by the first server to the web server over the Internet via a second device that is addressed by the selected first IP address, and wherein the receiving by the first server from the web server comprises receiving, by the first server from the web server via the second device.

### Claim 4 (depends on 2)

The method according to claim 2, wherein the list of IP addresses is stored in the first server.

### Claim 5 (depends on 2)

The method according to claim 2, further comprising determining, by the first server, that the first response received by the first server is a proper response.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the sending, by the first server to the first device of the first response is in response to the determining that the first response received by the first server is a proper response.

### Claim 7 (depends on 2)

The method according to claim 2, wherein the list of IP addresses is stored in a second server that is distinct from the first server.

### Claim 8 (depends on 7)

The method according to claim 7, further comprising sending, by the second server to the first server over the Internet, to be stored thereon, the list.

### Claim 9 (depends on 2)

The method according to claim 2, wherein the selecting of the first IP address is based on, or uses, load balancing.

### Claim 10 (depends on 2)

The method according to claim 2, wherein the selecting of the first IP address is based on, or uses, random selection.

### Claim 11 (depends on 10)

The method according to claim 10, wherein random selection uses, or is based on, one or more random numbers generated by a random number generator.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the random number generator is hardware based.

### Claim 13 (depends on 12)

The method according to claim 12, wherein the random number generator uses thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena.

### Claim 14 (depends on 11)

The method according to claim 11, wherein the random number generator is software based.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the random number generator is based on executing an algorithm for generating pseudo-random numbers.

### Claim 16 (depends on 1)

The method according to claim 1, wherein the first device is, in part or entirely, integrated within an appliance.

### Claim 17 (depends on 16)

The method according to claim 16, wherein a primary functionality of the appliance is associated with food storage, handling, or preparation.

### Claim 18 (depends on 16)

The method according to claim 16, wherein a primary function of the appliance is associated with environmental control, and the appliance consists of, or is part of, an HVAC system.

### Claim 19 (depends on 16)

The method according to claim 16, wherein a primary function of the appliance is associated with cleaning, wherein the primary function is associated with clothes cleaning, and the appliance is a washing machine or a clothes dryer, or wherein the appliance is a vacuum cleaner.

### Claim 20 (depends on 16)

The method according to claim 16, wherein a primary function of the appliance is associated with water control or water heating.

### Claim 21 (depends on 16)

The method according to claim 16, wherein the appliance is an answering machine, a telephone set, a home cinema method, a HiFi method, a CD or DVD player, an electric furnace, a trash compactor, a smoke detector, a light fixture, or a dehumidifier.

### Claim 22 (depends on 16)

The method according to claim 16, wherein the appliance is a battery-operated portable electronic device, and the appliance is a notebook, a laptop computer, a media player, a cellular phone, a Personal Digital Assistant (PDA), an image processing device, a digital camera, a video recorder, or a handheld computing device.

### Claim 23 (depends on 1)

The method according to claim 1, wherein the first device comprises, is integrated with, or is part of, a wearable device that is wearable on a person.

### Claim 24 (depends on 2)

The method according to claim 2, wherein the selecting of the first IP address is based on, or uses, an estimated geographical location of the respective device or of the web server.

### Claim 25 (depends on 24)

The method according to claim 24, further comprising estimating a geographical location of the first device or of the web server, using geolocation.

### Claim 26 (depends on 25)

The method according to claim 25, wherein the geolocation is based on IP geolocation.

### Claim 27 (depends on 25)

The method according to claim 25, wherein the geolocation is based on W3C Geolocation Application Programming Interface (API).

### Claim 28 (depends on 24)

The method according to claim 24, wherein the selecting of the first IP address is based on estimated as being geolocated in the same area as the first device or the web server.

### Claim 29 (depends on 28)

The method according to claim 28, wherein the selecting of the first IP address is based on estimated as being in the same continent, country, state, region, city, postal/zip code, latitude, longitude, or Timezone as the first device or the web server.

### Claim 30 (depends on 2)

The method according to claim 2, wherein the selecting of the first IP address is based on being a recent one to be selected, or based on being the least recent to be selected.

### Claim 31 (depends on 2)

The method according to claim 2, wherein the selecting of the first IP address is based on manually selecting by a user.

### Claim 32 (depends on 31)

The method according to claim 31, further comprising displaying, to the user, at least part of IP addresses in the list, and selecting, by the user, the first IP address from the list.

### Claim 33 (depends on 2)

The method according to claim 2, wherein a number of IP addresses in the list is at least 10, 20, 50, 100, 200, 500, 1,000, 2,000, 5,000, 10,000, 20,000, 50,000, 100,000, 200,000, 500,000, 1,000,000, 2,000,000, 5,000,000, or 10,000,000.

### Claim 34 (depends on 1)

The method according to claim 1, further comprising using, by the first device, the received first response, in response to the determining that the received first response is a proper response.

### Claim 35 (depends on 1)

The method according to claim 1, wherein the content comprises, or consists of, a web-page or a web-site that includes, consists of, or comprises, audio data, voice data, multimedia data, video data, or a computer program.

### Claim 36 (depends on 1)

The method according to claim 1, wherein the first server consists of, includes, is part of, or is integrated with, a proxy server.

### Claim 37 (depends on 36)

The method according to claim 36, wherein the proxy server consists of, includes, is part of, or is integrated with, an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a SOCKS proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof.

### Claim 38 (depends on 1)

The method according to claim 1, wherein the determining of the received first or second response comprises identifying and checking a HTTP status code that is received as part of the respective response.

### Claim 39 (depends on 38)

The method according to claim 38, wherein the respective response is determined as a proper response in response to a status code of 2xx.

### Claim 40 (depends on 39)

The method according to claim 39, wherein the respective response is determined as not being a proper response responsive to a status code of 4xx or 5xx.

### Claim 41 (depends on 40)

The method according to claim 40, wherein the respective response is determined as not being a proper response responsive to a status code of HTTP 404 error message.

### Claim 42 (depends on 38)

The method according to claim 38, wherein the checking of the respective response comprises using a timeout mechanism.

### Claim 43 (depends on 42)

The method according to claim 42, wherein the respective response is determined as not being a proper response in response to not receiving a proper response after elapsed defined time period after a sending of the respective request.

### Claim 44 (depends on 38)

The method according to claim 38, wherein the respective response is determined as not being a proper response in response to detecting a redirection.

### Claim 45 (depends on 44)

The method according to claim 44, wherein the redirection is identified by checking that the HTTP status code is 3xx Redirection.

### Claim 46 (depends on 38)

The method according to claim 38, for use with a criterion, wherein the determining comprises verifying if the respective response received satisfies the criterion.

### Claim 47 (depends on 46)

The method according to claim 46, wherein the criterion relates to a feature, characteristic, or type, of the received response.

### Claim 48 (depends on 46)

The method according to claim 46, wherein the criterion comprises a value, and wherein the respective response is determined as not being a proper response in response to comparing a content feature, characteristic, or type, to the value.

### Claim 49 (depends on 48)

The method according to claim 48, wherein the criterion comprises a value of a size of a file, and wherein the response is determined as not being a proper response in response to comparing the received response size to the value.

### Claim 50 (depends on 1)

The method according to claim 1, wherein the first device comprises, or is part of, a client device in a client/server architecture.

### Claim 51 (depends on 50)

The method according to claim 50, wherein the client device is housed in a single enclosure that is a hand-held enclosure or a portable enclosure.

### Claim 52 (depends on 50)

The method according to claim 50, wherein the client device consists of, comprises, is part of, or is integrated with, a notebook computer, a laptop computer, a media player, a Digital Still Camera (DSC), a Digital video Camera (DVC or digital camcorder), a Personal Digital Assistant (PDA), a cellular telephone, a digital camera, a video recorder, or a smartphone.

### Claim 53 (depends on 50)

The method according to claim 50, wherein the client device consists of, comprises, is part of, or is integrated with, a smartphone that comprises, or is based on, an Apple iPhone 6 or a Samsung Galaxy S6.

### Claim 54 (depends on 50)

The method according to claim 50, further comprising storing, operating, or using, by the client device, an operating system.

### Claim 55 (depends on 54)

The method according to claim 54, wherein the operating system is a mobile operating system.

### Claim 56 (depends on 55)

The method according to claim 55, wherein the mobile operating system comprises Android version 2.2 (Froyo), Android version 2.3 (Gingerbread), Android version 4.0 (Ice Cream Sandwich), Android Version 4.2 (Jelly Bean), Android version 4.4 (KitKat), Apple iOS version 3, Apple iOS version 4, Apple iOS version 5, Apple iOS version 6, Apple IOS version 7, Microsoft Windows® Phone version 7, Microsoft Windows® Phone version 8, Microsoft Windows® Phone version 9, or Blackberry® operating system.

### Claim 57 (depends on 1)

The method according to claim 1, further comprising storing, operating, or using, by the first device, a web browser.

### Claim 58 (depends on 57)

The method according to claim 57, wherein the web browser comprises of, or is based on, Microsoft Internet Explorer, Google Chrome, Opera™, or Mozilla Firefox®.

### Claim 59 (depends on 57)

The method according to claim 57, wherein the web browser is a mobile web browser.

### Claim 60 (depends on 59)

The method according to claim 59, wherein the mobile web browser comprises of, or is based on, Safari, Opera Mini™, or Android web browser.

### Claim 61 (depends on 57)

The method according to claim 57, further comprising returning an error message to the web browser.

### Claim 62 (depends on 57)

The method according to claim 57, wherein the identifying comprises identifying as part of an execution of the web browser.

### Claim 63 (depends on 62)

The method according to claim 62, wherein the identifying is performed as part of a plug-in or an extension integrated with the web browser.

### Claim 64 (depends on 63)

The method according to claim 63, wherein the identifying is performed by the web browser that is integrated a form of a plug-in or an extension.

### Claim 65 (depends on 63)

The method according to claim 63, wherein the identifying of the URL request uses a plug-in or an extension to the web browser.

### Claim 66 (depends on 63)

The method according to claim 63, wherein the integration is by hooking to the web browser, wherein the integration is in a filter driver form, or, wherein the web browser and the steps are communicating using an Inter-Process Communication (IPC).

### Claim 67 (depends on 66)

The method according to claim 66, wherein the identifying of the first URL request uses hooking to the web browser, a filter driver form, or using an Inter-Process Communication (IPC).

### Claim 68 (depends on 66)

The method according to claim 66, wherein the IPC is using a file sharing, a signal, a socket, a pipe, a message queue, a shared memory, a semaphore, or memory mapped file.

### Claim 69 (depends on 66)

The method according to claim 66, wherein the IPC is using a clipboard, a Component Object Model (COM), a data copy, a DDE protocol, or mailslots.
