Smart proxy rotator
Assigneemetacluster It, UAB
InventorsMartynas Juravicius, Eivydas Vilcinskas
Life of this patent
Estimating the term…
Abstract
The task, logic of HTTP/HTTPS session statistics interception and collection is moved to the client side instead of the proxy layer. Encrypted HTTPS tunnel is terminated at the client end, making the actual content or data in transit invisible to both proxies and the smart proxy rotator (SPR). Client's scraping software has a plug-in installed that expands its functionality. HTTP/HTTPS session quality metrics are intercepted and collected at the client side, then sent to the SPR. Proxy usage mark “can be used” is obtained from the SPR for the currently analyzed proxy, based on the results of metrics analysis.
Claims20
A non-transitory computer readable medium for identifying a threshold of at least one proxy provider, the medium comprising instructions that, when executed by a processor, direct the processor to:
- provide a web scraper with a first proxy from a first proxy provider;
- establish a first connection to a target server through the first proxy;
- obtain first content from the target server;
- accumulate performance and availability information of the first proxy;
- provide the web scraper with a second proxy from the first proxy provider;
- establish a second connection to a target server through the second proxy;
- obtain second content from the target server;
- accumulate performance and availability information of the second proxy;
- calculate and configure utilization thresholds of the first proxy provider by evaluating at least one of the performance and availability information of the first proxy or the performance and availability information of the second proxy.
A non-transitory computer readable medium for identifying thresholds of at least a first proxy provider and a second proxy provider, the medium comprising instructions that, when executed by a processor, direct the processor to:
- provide a web scraper with at least a first proxy of the first proxy provider;
- establish a first connection to a target server through the first proxy of the first proxy provider;
- obtain first content from the target server;
- accumulate performance and availability information of the first proxy of the first proxy provider;
- provide the web scraper with at least a first proxy of the second proxy provider;
- establish a second connection to a target server through the second proxy of the second proxy provider;
- obtain second content from the target server;
- accumulate performance and availability information of the second proxy of the second proxy provider;
- calculate and configure utilization the thresholds by evaluating the performance and availability information of the first proxy of the first proxy provider and the performance and availability information of the first proxy of the second proxy provider.
A computer-implemented method for identifying a threshold of at least one proxy provider, the method comprising:
- running a proxy rotator on a computing device connected to a network;
- utilizing the proxy rotator to obtain a first proxy from a first proxy provider;
- utilizing a web scraper to:
- establish a first connection to a target server through the first proxy;
- receive first content designated within a first request through the first connection;
- accumulate performance and availability information of the first proxy;
- utilizing the proxy rotator to obtain a second proxy from the first proxy provider;
- utilizing the web scraper to:
- establish a second connection to the target server through the second proxy;
- receive second content designated within a second request through the second connection;
- accumulate performance and availability information of the second proxy;
- providing the performance and availability information of the first proxy and the performance and availability information of the second proxy to the proxy rotator;
- utilizing the proxy rotator to calculate a weight for the first proxy provider based upon at least one of the performance and availability information of the first proxy or the performance and availability information of the second proxy;
- and
- identifying a threshold of the first proxy provider based on the weight calculated.
Dependencies are inferred from the claim text.