Optimizing scraping requests through browsing profiles

#In at least one aspect, the present disclosure relates to methods for optimizing the parameters of web scraping requests with the artefacts of human-like browsing behavior for obtaining the desired data while circumventing the blocking by the content owners.

#The following sections will elaborate and clarify the technology notions the solution employs, is dependent upon, or is related to.

#Web scraping (also known as screen scraping, data mining, web harvesting) in its most general sense is the automated gathering of data from the internet. More technically, it is the practice of gathering data from the internet through any means other than a human using a web browser or a program interacting with an application programming interface (API). Web scraping is usually accomplished by executing a program that queries a web server and requests data automatically, then parses the data to extract the requested information.

#The features and advantages of the example embodiments described herein will become apparent to those skilled in the art to which this disclosure relates upon reading the following description, with reference to the accompanying drawings, in which:

#FIG. 1 is an exemplary component diagram that shows the overall architecture of components and context of at least one aspect of solution functions.

#FIG. 2A demonstrates bulk creation of blank browsing profiles populated with a minimum set of static parameters.

#Some general terminology descriptions may be helpful and are included herein for convenience and are intended to be interpreted in the broadest possible interpretation. Elements that are not imperatively defined in the description should have the meaning as would be understood by a person skilled in the art. Elements 104 through 116 identify parts of the Service Provider Infrastructure, with elements 102, 120-126 and 140 showing external components or systems.

#User Device 102—can be any suitable user computing device including, but not limited to, a smartphone, a tablet computing device, a personal computing device, a laptop computing device, a gaming device, a vehicle infotainment device, a smart appliance (e.g., smart refrigerator or smart television), a cloud server, a mainframe, a notebook, a desktop, a workstation, a mobile device, or any other electronic device used for making a scraping request.

#Scraping Agent 104—is a component of the Service Provider Infrastructure 114 that is responsible for containing and running the scraping applications executing the scraping requests originating from the commercial users, as well as accepting said requests from the users.