Systems and methods for web scraping

A system comprising:

• a computer system comprising a first hardware processor, a memory, and a client web browser;
• an aggregator comprising a second hardware processor, wherein the aggregator receives a Hypertext Transfer Protocol (HTTP) request from the client web browser and generates a first instruction responsive to the HTTP request;
• a header manipulator comprising a third hardware processor, wherein the header manipulator modifies first header information in the first instruction to produce a first header-modified HTTP request;
• a proxy server comprising a fourth hardware processor, wherein the proxy server forwards the first header-modified HTTP request to a first target website and receives a first HTTP response from the first target website based on the first header-modified HTTP request;
• and
• a man in the middle (MITM) proxy comprising a hardware server, wherein the MITM proxy:
• receives the first HTTP response, wherein the first HTTP response comprises a webpage and an on-page JavaScript call configured to perform a first web browser function in the client web browser,
• modifies the first HTTP response by adding injected JavaScript data to produce a modified HTTP response,
• wherein the modified HTTP response alters a result of the on-page JavaScript call to provide a different perceived value of the first web browser function to the first target website to produce an anti-bot detection protocol for the first target website,
• wherein the first web browser function determines a number of plugins that are installed in the client web browser, and
• wherein the different perceived value is a number of plugins that is different from the number of plugins installed in the client web browser, and
• forwards the modified HTTP response to the aggregator,
• wherein the aggregator sends a result to the client web browser responsive to the modified HTTP response.