# US 12,182,219 B2 · Adaptive data collection optimization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,219 B2
- Document ID: US12182219B2
- Status: Granted
- Kind: B2
- Application number: 18/477,095
- Filed: 2023-09-28
- Published: 2024-01-25
- Granted: 2024-12-31
- Earliest claimed: 2021-08-24
- Assignees: Oxylabs, UAB
- Applicants: Oxylabs, Uab
- Inventors: Martynas Juravicius; Erikas Bulba; Mantas Briliauskas
- Web page: https://patentgrep.com/patent/US12182219B2

## CPC

- G06F 16/955: using information identifiers, e.g. uniform resource locators [URL]
- G06F 11/3409: {for performance assessment}
- G06F 16/906: Clustering; Classification
- G06F 16/951: Indexing; Web crawling techniques
- H04L 67/56: Provisioning of proxy services (store-and-forward switching systems in data switching networks H04L12/54)
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods to intelligently optimize data collection requests are disclosed. In one embodiment, systems are configured to identify and select a complete set of suitable parameters to execute the data collection requests. In another embodiment, systems are configured to identify and select a partial set of suitable parameters to execute the data collection requests. The present embodiments can implement machine learning algorithms to identify and select the suitable parameters according to the nature of the data collection requests and the targets. Moreover, the embodiments provide systems and methods to generate feedback data based upon the effectiveness of the data collection parameters. Furthermore, the embodiments provide systems and methods to score the set of suitable parameters based on the feedback data and the overall cost, which are then stored in an internal database.

## Claims

### Claim 1

A method to increase a quality of data extraction by a service provider infrastructure comprising:
receiving a scraping request to a target domain from a user device via a network;
collecting a set of scraping request parameters to execute the scraping request;
initiating a process of identifying and selecting the set of the scraping request parameters, with the set comprising either a full set, which includes an entirety of the scraping request parameters, or a partial set, which includes less than the entirety of the scraping request parameters,
wherein, the selecting of the set of the scraping request parameters is done by applying a machine learning algorithm;
selecting a proxy server serving the set of the scraping request parameters;
using the set of the scraping request parameters and the selected proxy server to implement the scraping request to the target domain; and,
receiving from the target domain through the proxy server, response data to the scraping request.

### Claim 2 (depends on 1)

The method of claim 1, wherein the response data from the scraping request comprises a ‘block’ response or a ‘non-block’ response.

### Claim 3 (depends on 2)

The method of claim 2, wherein, if the ‘block’ response for the scraping request is received, the method to increase a quality of data extraction by the service provided infrastructure is repeated with a new set of scraping request parameters.

### Claim 4 (depends on 3)

The method of claim 3, wherein the method to increase a quality of data extraction by the service provided infrastructure is repeated for the scraping request until the ‘non-block’ response or a maximum threshold of attempts is reached.

### Claim 5 (depends on 2)

The method of claim 2, wherein data from the ‘non-block’ response is used for future scraping actions.

### Claim 6 (depends on 2)

The method of claim 2, wherein when the response data comprises a ‘non-block’ response the response is sent to the user device via a network.

### Claim 7 (depends on 2)

The method of claim 2, wherein the set of the scraping request parameters are scored and the set of scraping request parameters receiving the ‘non-block’ response receives a higher score than the set of the scraping request parameters receiving the ‘block’ response.

### Claim 8 (depends on 1)

The method of claim 1, wherein when the selecting comprises the full set of the scraping request parameters, every parameter accompanying the URL of target received in the scraping request is amended to a set of suitable parameters to execute the scraping request.

### Claim 9 (depends on 1)

The method of claim 1, wherein when the selecting comprises the partial set of the scraping request parameters, one or more specific parameters accompanying the URL of target are accepted while other parameters are disregarded.

### Claim 10 (depends on 1)

The method of claim 1, wherein the set of scraping request parameters is affected by an amount of overall scraping request cost calculated for the scraping request parameters.

### Claim 11 (depends on 1)

The method of claim 1, wherein the machine learning algorithm is modified as new scraping request parameters are recorded within different sets of parameters.

### Claim 12 (depends on 11)

The method of claim 11, wherein the scraping request parameters of a particular set are based on a specific machine learning algorithm.

### Claim 13 (depends on 11)

The method of claim 11, wherein the particular set of suitable parameters are stored in a database coupled with the particular set of suitable parameters.

### Claim 14 (depends on 1)

The method of claim 1, wherein the partial set comprises a single scraping request parameter or comprises a combination of any of:
proxy type;
proxy location;
proxy ASN (Autonomous System Number);
operating system preference;
browser preference;
conditions for headers;
Hypertext Transfer Protocol (HTTP) protocol type and version.

### Claim 15 (depends on 1)

The method of claim 1, wherein the type of machine learning algorithm is determined by service provider infrastructure.

### Claim 16 (depends on 1)

The method of claim 1 wherein the method is repeated with at least one subsequent scraping request, and, wherein the scraping request and the at least one subsequent scraping request form a plurality of scraping requests.

### Claim 17 (depends on 16)

The method of claim 16 wherein the machine learning algorithm is modified according to the response data received from the plurality of the scraping requests.

### Claim 18 (depends on 1)

The method of claim 1, further comprising:
scoring the set of scraping request parameters to form a scored set of scraping request parameters;
wherein the scoring of the set of scraping request parameters is affected by an amount of overall scraping request cost calculated for the scraping request parameters.