Smart proxy rotator
AssigneeMetacluster Lt, Uab
InventorsMartynas Juravicius, Eivydas Vilcinskas
Life of this patent
Estimating the term…
Abstract
The task, logic of HTTP/HTTPS session statistics interception and collection is moved to the client side instead of the proxy layer. Encrypted HTTPS tunnel is terminated at the client end, making the actual content or data in transit invisible to both proxies and the smart proxy rotator (SPR). Client's scraping software has a plug-in installed that expands its functionality. HTTP/HTTPS session quality metrics are intercepted and collected at the client side, then sent to the SPR. Proxy usage mark “can be used” is obtained from the SPR for the currently analyzed proxy, based on the results of metrics analysis.
Claims20
A computer-implemented method for dynamically configuring a utilization threshold of a proxy provider, the method comprising:
- running a smart proxy rotator (SPR) on a computing device connected to a network;
- obtaining a proxy from the SPR, according to currently valid SPR operational settings, by a Web Scraper to use for a request for content;
- establishing, by the Web Scraper, an encrypted connection to a target server through the proxy;
- requesting and obtaining the content designated within the request, through the encrypted connection, by the Web Scraper;
- providing, by the Web Scraper, performance and availability information of the proxy to the SPR;
- calculating, at the SPR, a weight for the proxy provider;
- and
- dynamically configuring utilization threshold of the proxy provider based on the calculation.
A non-transitory computer readable medium computer medium for dynamically configuring a utilization threshold of a proxy provider comprising instructions which, when executed by a computing device, causes the computing device to:
- provide a Web Scraper with a proxy from a proxy provider list;
- accumulate performance and availability information about the proxy provided by the Web Scraper after the proxy has been used to obtain content from a target, wherein the content is unreadable to a Smart Proxy Rotator (SPR) due to network transport layer or application layer encryption;
- calculate and configure utilization thresholds of a plurality of proxy providers by evaluating proxy performance and availability information received from the Web Scraper.
A system comprising:
- a Smart Proxy Rotator (SPR) operable to provide a plurality of proxies over a network;
- and
- a Web Scraper operable to obtain a first proxy of the plurality of proxies from the SPR according to currently valid SPR operational settings, to establish an encrypted connection with the target server through the first proxy, to request content from a target server, to receive the content designated within the request through the encrypted connection, and to provide performance and availability information of the first proxy to the SPR,
- wherein the SPR is operable to calculate a weight for the proxy provider of the first proxy based on first proxy performance statistics analysis, and to dynamically configure utilization threshold of the proxy provider based on the weight calculated.
Dependencies are inferred from the claim text.