# US 12,010,101 B2 · System and method for managing non-direct URL fetching service

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,010,101 B2
- Document ID: US12010101B2
- Status: Granted
- Kind: B2
- Application number: 17/862,457
- Filed: 2022-07-12
- Published: 2022-11-10
- Granted: 2024-06-11
- Earliest claimed: 2019-04-02
- Assignees: Bright Data Ltd.
- Applicants: Bright Data Ltd.
- Inventors: Derry Shribman; Ofer Vilenski
- Web page: https://patentgrep.com/patent/US12010101B2

## CPC

- G06F 16/9574: {of access to content, e.g. by caching}
- H04L 63/0272: {Virtual private networks}
- G06F 16/955: using information identifiers, e.g. uniform resource locators [URL]
- G06F 16/9566: {URL specific, e.g. using aliases, detecting broken or misspelled links}
- H04L 63/029: {Firewall traversal, e.g. tunnelling or, creating pinholes}
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/56: Provisioning of proxy services (store-and-forward switching systems in data switching networks H04L12/54)
- H04L 67/06: specially adapted for file transfer, e.g. file transfer protocol [FTP]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A method for providing and managing non-direct URL fetching service for retrieving a content from a web server to a client device is disclosed, such as for overcoming geo-blocking or a Man-In-The-Middle (MITM) attack. The non-direct fetching method may use intermediate devices, such as proxy server, Data-Center proxy server, tunnel devices, or any combination thereof. A URL request may be sent in parallel using both direct and non-direct fetching schemes, in order to verify the need for using the non-direct fetching service. Direct or non-direct fetching scheme may be selected by using a file that associates a fetching scheme to the requested URL. The selection of the fetching mechanism may use dynamically in real-time updating of a Proxy Auto-Configuration (PAC) file. As part of an accounting scheme, quotas may be applied to a cumulative received data or a time duration of using a non-direct fetching service.

## Claims

### Claim 1

A method for fetching, by a client device, a web-page that is identified by a Uniform Resource Locator (URL) and stored in a web server, and for identifying and overcoming blocking, the method comprising:
identifying, as part of executing of a web browser at the client device, the web-page;
sending, by the client device to the web server over the Internet, a first Hypertext Transfer Protocol (HTTP) request that includes the URL and includes an Internet Protocol (IP) of the client device as a source address, in response to the identifying of the web-page, so that the IP address of the client device is available to the web server;
receiving, by the client device from the web server over the Internet, a first response, in response to the sending of the first HTTP request;
before receiving of the first response, sending, by the client device over the Internet to a first device that is distinct from the web server, a second HTTP request that includes the URL;
receiving, by the client device from the first device over the Internet, a second response, in response to the sending of the second HTTP request;
determining, by the client device, whether the first response is a proper response;
in response to the determining that the first response is a proper response, using, as part of executing of a web browser at the client device, the first response, responsive to the identifying;
in response to the determining that the first response is not a proper response, determining, by the client device, that the second response is a proper response or a failed response; and
in response to the determining that the first response is not a proper response and that the second response is a proper response, using, as part of executing of a web browser at the client device, the second response, responsive to the identifying.

### Claim 2 (depends on 1)

The method according to claim 1, further for overcoming geo-blocking, Internet censorship, Man-In-the-Middle (MITM) attack, or firewalling.

### Claim 3 (depends on 1)

The method according to claim 1, further comprising comparing the first and second responses, and identifying the blocking based on the comparing.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising:
sending, by the first device to the web server over the Internet, a third HTTP request that that includes the URL, in response to a receiving of the second HTTP request from the client device;
receiving, by the first device from the web server over the Internet, the second response, in response to the sending of the third HTTP request; and
sending, by the first device to the client device, the received second response,
wherein the third HTTP request does not include the IP address of the client device so that the IP address of the client device is unknown to the web server.

### Claim 5 (depends on 1)

The method according to claim 1, for use with a list of URLs, further comprising checking the URL of the identified web-page for determining whether the URL is included in the list, wherein the sending of the second HTTP request to the first device is in response to the determining that the identified URL is included or not included in the list.

### Claim 6 (depends on 5)

The method according to claim 5, further comprising adding the URL to the list in response to the determining that the first response is not a proper response.

### Claim 7 (depends on 5)

The method according to claim 5, further comprising adding the URL to the list in response to the determining that the first response is not a proper response and that the second response is a proper response.

### Claim 8 (depends on 5)

The method according to claim 5, wherein the list is stored in the client device.

### Claim 9 (depends on 5)

The method according to claim 5, wherein the list is stored in a second device that is distinct from the client device.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the list is stored in a server device.

### Claim 11 (depends on 9)

The method according to claim 9, further comprising:
sending, by the client device to the second device over the Internet, the URL of the identified web-page;
determining, by the second device, whether the URL is included in the list; and
sending, by the second device to the first device, an indication of the determining of whether the URL is included in the list.

### Claim 12 (depends on 9)

The method according to claim 9, further comprising receiving and storing, by the client device from the second device, the list.

### Claim 13 (depends on 1)

The method according to claim 1, further comprising notifying a user in response to the determining that the first response is a proper response.

### Claim 14 (depends on 1)

The method according to claim 1, further comprising notifying a user in response to the determining that the first response is not a proper response.

### Claim 15 (depends on 1)

The method according to claim 1, further comprising notifying a user in response to the determining that the second response is a proper response.

### Claim 16 (depends on 1)

The method according to claim 1, further comprising notifying a user in response to the determining that the second response is not a proper response.

### Claim 17 (depends on 1)

The method according to claim 1, wherein the sending by the client device the first and second HTTP requests are performed at least in part in parallel, and wherein the parallel operation is based on, or uses, multitasking or multiprocessing.

### Claim 18 (depends on 1)

The method according to claim 1, wherein the web server and the client device are located in different geographical locations.

### Claim 19 (depends on 18)

The method according to claim 18, wherein the different locations are two continents, two countries, two states, two regions, two cities, two streets, two ZIP codes, or two timezones.

### Claim 20 (depends on 18)

The method according to claim 18, wherein the first device is located in the same location as the client device.

### Claim 21 (depends on 18)

The method according to claim 18, wherein the second device is located in the same location as the web server.

### Claim 22 (depends on 18)

The method according to claim 18, wherein the second device is located in a location that is different from the web server location and from the client device location.

### Claim 23 (depends on 1)

The method according to claim 1, further comprising identifying, by the client device, a geographical location of the client device, the first device, the web server, or any combination thereof, using geolocation.

### Claim 24 (depends on 23)

The method according to claim 23, wherein the geolocation is based on IP geolocation.

### Claim 25 (depends on 24)

The method according to claim 24, wherein the IP geolocation is based on W3C Geolocation Application Programming Interface (API).

### Claim 26 (depends on 1)

The method according to claim 1, wherein the determining whether the first or second response is a proper response comprises identifying and checking a HTTP status code that is received in the respective first or second response.

### Claim 27 (depends on 26)

The method according to claim 26, wherein the respective response is determined as a proper response responsive to a status code of 2xx.

### Claim 28 (depends on 26)

The method according to claim 26, wherein the responsive response is determined as not a proper response responsive to a status code of 4xx or 5xx.

### Claim 29 (depends on 26)

The method according to claim 26, wherein the respective response is determined as not a proper response responsive to a status code of HTTP 404 error message.

### Claim 30 (depends on 1)

The method according to claim 1, wherein the determining whether the first or second response is a proper response comprises using a timeout mechanism.

### Claim 31 (depends on 30)

The method according to claim 30, wherein the respective response is determined not a proper response in response to not receiving any response after elapsed defined time period after the sending of a respective HTTP request.

### Claim 32 (depends on 1)

The method according to claim 1, wherein the determining whether the first or second response is a proper response comprises checking if an URL redirection is identified in the respective response, and wherein the respective response is determined as not a proper response responsive to detecting the URL redirection.

### Claim 33 (depends on 32)

The method according to claim 32, wherein the URL redirection is identified by checking that a respective HTTP status code is 3xx Redirection.

### Claim 34 (depends on 1)

The method according to claim 1, for use with a criterion, wherein the determining whether the first or second response is a proper response comprises verifying if the respective response satisfies the criterion.

### Claim 35 (depends on 34)

The method according to claim 34, wherein the criterion is associated with a feature, characteristic, or type, of the respective response, wherein the criterion comprises a value, and wherein the respective response is determined as not proper in response to comparing the respective response feature, characteristic, or type, to the value.

### Claim 36 (depends on 35)

The method according to claim 35, wherein the criterion comprises a value of a size of a file, and wherein the respective response is determined as not proper in response to comparing the respective response size to the value.

### Claim 37 (depends on 36)

The method according to claim 36, for use with a group of multiple devices that includes the first device, wherein the method further comprising selecting of the first device from the group.

### Claim 38 (depends on 37)

The method according to claim 37, wherein the selecting is based on, or uses, load balancing.

### Claim 39 (depends on 37)

The method according to claim 37, wherein the selecting is based on, or uses, random selection.

### Claim 40 (depends on 39)

The method according to claim 39, wherein the random selection uses, or is based on, one or more random numbers generated by a random number generator.

### Claim 41 (depends on 40)

The method according to claim 40, wherein the random number generator is hardware based.

### Claim 42 (depends on 41)

The method according to claim 41, wherein the random number generator uses, or is based on, thermal noise, shot noise, nuclear decaying radiation, photoelectric effect, or quantum phenomena.

### Claim 43 (depends on 40)

The method according to claim 40, wherein the random number generator is software based.

### Claim 44 (depends on 43)

The method according to claim 43, wherein the random number generator uses, or is based on, executing an algorithm for generating pseudo-random numbers.

### Claim 45 (depends on 37)

The method according to claim 37, wherein the selecting is based on, or uses, an estimated geographical location of the first device, the client device, the web server, or any combination thereof.

### Claim 46 (depends on 45)

The method according to claim 45, further comprising estimating the geographical location using geolocation.

### Claim 47 (depends on 46)

The method according to claim 46, wherein the geolocation is based on, or uses, an IP geolocation.

### Claim 48 (depends on 46)

The method according to claim 46, wherein the geolocation is based on, or uses, W3C Geolocation Application Programming Interface (API).

### Claim 49 (depends on 45)

The method according to claim 45, wherein the first device is selected based on being estimated to be located in the same area as the client device, the web server, or any combination thereof.

### Claim 50 (depends on 49)

The method according to claim 49, wherein the first device is selected based on being estimated to be located in the same continent, country, state, region, city, postal/zip code, latitude, longitude, or Timezone as the client device, the web server, or any combination thereof.

### Claim 51 (depends on 37)

The method according to claim 37, wherein the first device is selected based on being a recent one to be selected, or based on being a least recent to be selected.

### Claim 52 (depends on 37)

The method according to claim 37, wherein the first device is manually selected by a user.

### Claim 53 (depends on 52)

The method according to claim 52, further comprising displaying to the user the multiple devices in the group, and selecting, by the user, the first device from the group.

### Claim 54 (depends on 37)

The method according to claim 37, wherein selecting is performed by the client device.

### Claim 55 (depends on 37)

The method according to claim 37, wherein selecting is performed by a device that is distinct from the client device.

### Claim 56 (depends on 55)

The method according to claim 55, wherein the sending, by the client device to the first device over the Internet, of the second HTTP request, comprises:
sending, by the client device to a server device over the Internet, the second HTTP request;
receiving, by the server device from the client device, the second HTTP request;
selecting, by the server device, the first device from the group, in response to the receiving of the second HTTP request;
sending, by the server device to the selected first device over the Internet, a third HTTP request that includes the URL, in response to the selecting;
receiving, by the selected first device from the server device, the sent third HTTP request; and
sending, by the first device to the web server over the Internet, the URL, in response to the receiving of the third HTTP request.

### Claim 57 (depends on 56)

The method according to claim 56, wherein the receiving, by the client device from the first device over the Internet, of the second response, comprises:
receiving, by the first device from the web server over the Internet, the second response, in response to the sending of the URL;
sending, by the first device to the server device over the Internet, the received second response, in response to the receiving of the second response;
receiving, by the server device from the first device, the sent second response; and
sending, by the server device to the client device over the Internet, the received second response.

### Claim 58 (depends on 57)

The method according to claim 57, further for use with a virtualization, wherein the client device or the first device comprises a server device that virtualize the respective client device or first device.

### Claim 59 (depends on 58)

The method according to claim 58, wherein the virtualization is executed as part of a Virtual Machine (VM).

### Claim 60 (depends on 59)

The method according to claim 59, for use with a host computer that implement the VM, wherein the method further comprising executing, by the host computer, a hypervisor or a Virtual Machine Monitor (VMM).

### Claim 61 (depends on 59)

The method according to claim 59, wherein the virtualization includes, is based on, or uses, full virtualization, para-virtualization, or hardware assisted virtualization.

### Claim 62 (depends on 36)

The method according to claim 36, wherein the first device comprises a proxy server.

### Claim 63 (depends on 62)

The method according to claim 62, wherein the proxy server comprises an HTTP proxy server, a web-proxy server, a caching proxy, an open-source caching proxy server, a cloud-based proxy server, an open proxy server, a forwarding proxy server, a reverse proxy server, a transparent proxy server, a non-transparent proxy server, an anonymous proxy server, a translation proxy server, a SOCKS proxy server, a CGI web proxy server, a suffix proxy server, an I2P anonymous proxy server, a DNS proxy server, or any combination thereof.

### Claim 64 (depends on 1)

The method according to claim 1, further comprising storing, operating, or using, by the client device, a client operating system.

### Claim 65 (depends on 64)

The method according to claim 64, wherein the client operating system comprises, or is based on, one out of Microsoft Windows 7, Microsoft Windows XP, Microsoft Windows 8, Microsoft Windows 8.1, Linux, and Google Chrome OS.

### Claim 66 (depends on 64)

The method according to claim 64, wherein the client operating system comprises a Real-Time Operating System (RTOS).

### Claim 67 (depends on 66)

The method according to claim 66, wherein the RTOS comprises FreeRTOS, SafeRTOS, QNX, VxWorks, or Micro-Controller Operating Systems (μC/OS).

### Claim 68 (depends on 1)

The method according to claim 1, further comprising storing, operating, or using, by the client device, the web browser.

### Claim 69 (depends on 68)

The method according to claim 68, wherein the web browser comprises of, or is based on, Microsoft Internet Explorer, Google Chrome, Opera™, or Mozilla Firefox®.

### Claim 70 (depends on 68)

The method according to claim 68, wherein the web browser is a mobile web browser.

### Claim 71 (depends on 70)

The method according to claim 70, wherein the mobile web browser comprises, or is based on, Safari, Opera Mini™, or Android web browser.

### Claim 72 (depends on 1)

The method according to claim 1, wherein at least part of steps of claim 1 are included in a Software Development Kit (SDK) that is provided as a non-transitory computer readable medium that contains computer instructions, and wherein the method further comprising installing the SDK.

### Claim 73 (depends on 1)

The method according to claim 1, wherein the client device or the first device comprises a wearable device that is wearable on a person.

### Claim 74 (depends on 73)

The method according to claim 73, wherein the wearable device is wearable on an organ of the person head.

### Claim 75 (depends on 74)

The method according to claim 74, wherein the organ is an eye, ear, face, cheek, nose, mouth, lip, forehead, or chin.

### Claim 76 (depends on 73)

The method according to claim 73, wherein the wearable device is constructed to have a form substantially similar to, is constructed to have a shape allowing mounting or wearing identical or similar to, or is constructed to have a form to at least in part substitute for, headwear, eyewear, or earpiece.

### Claim 77 (depends on 76)

The method according to claim 76, wherein the headwear structured as, or comprises, a bonnet, a cap, a crown, a fillet, a hair cover, a hat, a helmet, a hood, a mask, a turban, a veil, or a wig.

### Claim 78 (depends on 76)

The method according to claim 76, wherein the eyewear structured as, or comprises, glasses, sunglasses, a contact lens, a blindfold, or a goggle.

### Claim 79 (depends on 76)

The method according to claim 76, wherein the earpiece structured as, or comprises, a hearing aid, a headphone, a headset, or an earplug.

### Claim 80 (depends on 73)

The method according to claim 73, wherein the wearable device is shaped for permanently or releasably attachable to, or be part of, a clothing piece of a person.

### Claim 81 (depends on 80)

The method according to claim 80, wherein the attaching uses taping, gluing, pinning, enclosing, encapsulating, a pin, or a latch and hook clip.

### Claim 82 (depends on 81)

The method according to claim 81, wherein the clothing piece is a top, bottom, or full-body underwear, or a headwear, a footwear, an accessory, an outwear, a suit, a dress, a skirt, or a top.

### Claim 83 (depends on 73)

The method according to claim 73, wherein the wearable device further comprises an annular member defining an aperture therethrough that is sized for receipt therein of a part of a human body.

### Claim 84 (depends on 1)

The method according to claim 1, wherein at least one of the steps is performed by an application that is integrated with the web browser in a form of a plug-in or an extension.

### Claim 85 (depends on 84)

The method according to claim 84, wherein at least one of the steps performed by the client device is performed by an application that is integrated with the web browser in a form of a plug-in or an extension.

### Claim 86 (depends on 84)

The method according to claim 84, wherein the identifying of the web-page uses a plug-in or an extension to the web browser.

### Claim 87 (depends on 84)

The method according to claim 84, wherein the integration is by hooking to the web browser, wherein the integration is in a filter driver form, or, wherein the web browser and the steps are communicating using an Inter-Process Communication (IPC).

### Claim 88 (depends on 87)

The method according to claim 87, wherein the identifying of the web-page uses hooking to the web browser, a filter driver form, or using an Inter-Process Communication (IPC).

### Claim 89 (depends on 87)

The method according to claim 87, wherein the IPC is using a file sharing, a signal, a socket, a pipe, a message queue, a shared memory, a semaphore, or memory mapped file.

### Claim 90 (depends on 87)

The method according to claim 87, wherein the IPC is using a clipboard, a Component Object Model (COM), a data copy, a DDE protocol, or mailslots.

### Claim 91 (depends on 1)

The method according to claim 1, wherein the client device or the first device is integrated, in part or entirely, in an appliance.

### Claim 92 (depends on 91)

The method according to claim 91, wherein a primary functionality of the appliance is associated with food storage, handling, or preparation.

### Claim 93 (depends on 92)

The method according to claim 92, wherein a primary function of the appliance is heating food, and wherein the appliance is a microwave oven, an electric mixer, a stove, an oven, or an induction cooker.

### Claim 94 (depends on 92)

The method according to claim 92, wherein the appliance is a refrigerator, a freezer, a food processor, a dishwasher, a food blender, a beverage maker, a coffeemaker, or an iced-tea maker.

### Claim 95 (depends on 91)

The method according to claim 91, wherein a primary function of the appliance is associated with environmental control, and wherein the appliance is part of an HVAC system.

### Claim 96 (depends on 95)

The method according to claim 95, wherein a primary function of the appliance is associated with temperature control, and wherein the appliance is an air conditioner or a heater.

### Claim 97 (depends on 91)

The method according to claim 91, wherein a primary function of the appliance is associated with cleaning, wherein the primary function is associated with clothes cleaning, and the appliance is a washing machine or a clothes dryer, or wherein the appliance is a vacuum cleaner.

### Claim 98 (depends on 91)

The method according to claim 91, wherein a primary function of the appliance is associated with water control or water heating.

### Claim 99 (depends on 91)

The method according to claim 91, wherein the appliance is an answering machine, a telephone set, a home cinema method, a HiFi method, a CD or DVD player, an electric furnace, a trash compactor, a smoke detector, a light fixture, or a dehumidifier.

### Claim 100 (depends on 91)

The method according to claim 91, wherein the appliance is a battery-operated portable electronic device, and the appliance is a notebook, a laptop computer, a media player, a cellular phone, a Personal Digital Assistant (PDA), an image processing device, a digital camera, a video recorder, or a handheld computing device.

### Claim 101 (depends on 91)

The method according to claim 91, wherein the integration involves sharing a component.

### Claim 102 (depends on 101)

The method according to claim 101, wherein the integration involves housing in same enclosure, sharing same processor, or mounting onto same surface.

### Claim 103 (depends on 101)

The method according to claim 101, wherein the integration involves sharing a same connector.

### Claim 104 (depends on 103)

The method according to claim 103, wherein the connector is a power connector for connecting to a power source, and wherein the integration involves sharing the same connector for being powered from same power source, or wherein the integration involves sharing same power supply.

### Claim 105 (depends on 1)

The method according to claim 1, wherein the client device or the first device is housed in a single enclosure that is a hand-held enclosure or a portable enclosure.

### Claim 106 (depends on 105)

The method according to claim 105, wherein the client device is further integrated with at least one of a notebook computer, a laptop computer, a media player, a Digital Still Camera (DSC), a Digital video Camera (DVC or digital camcorder), a Personal Digital Assistant (PDA), a cellular telephone, a digital camera, a video recorder, or a smartphone.

### Claim 107 (depends on 106)

The method according to claim 106, wherein the smartphone comprises, or is based on, an Apple iPhone 6 or a Samsung Galaxy S6.

### Claim 108 (depends on 1)

The method according to claim 1, further comprising storing, operating, or using, by the client device or the first device, an operating system.

### Claim 109 (depends on 108)

The method according to claim 108, wherein the operating system is a mobile operating system.

### Claim 110 (depends on 109)

The method according to claim 109, wherein the mobile operating system comprises Android version 2.2 (Froyo), Android version 2.3 (Gingerbread), Android version 4.0 (Ice Cream Sandwich), Android Version 4.2 (Jelly Bean), Android version 4.4 (KitKat), Apple iOS version 3, Apple iOS version 4, Apple iOS version 5, Apple iOS version 6, Apple iOS version 7, Microsoft Windows® Phone version 7, Microsoft Windows® Phone version 8, Microsoft Windows® Phone version 9, or Blackberry® operating system.

### Claim 111 (depends on 1)

A non-transitory computer readable medium containing computer instructions that, when executed by a computer processor, cause the processor to perform all of the steps of claim 1.