Token-based authentication for a proxy web scraping service

A computer-implemented method for determining which servers are available to process web scraping jobs, comprising:

• obtaining, by a request intake manager, a first token indicating an identity of a client associated with credentials received with a first application programming interface (API) request from a client computing device, wherein the first API request specifies a target website to scrape and the first token is generated by an authentication service when the credentials are determined to be valid;
• selecting, by the request intake manager, a database server to store, in a job database that stores statuses of web scraping jobs while the web scraping jobs are being executed, a job description generated based on the first API request from a plurality of database servers identified as enabled in a database server status table, wherein the plurality of database servers operate independently of one another and each database server is configured to manage data storage to at least a portion of the job database;
• assigning, by the request intake manager, a job identifier to the job description, wherein the job identifier encodes an identifier for the selected database server;
• transmitting, by the request intake manager, the job identifier to the client computing device;
• and
• transmitting, by the request intake manager, the job identifier and the job description to the selected database server.

A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:

• obtaining, by a request intake manager, a first token indicating an identity of a client associated with credentials received with a first application programming interface (API) request from a client computing device, wherein the first API request specifies a target website to scrape and the first token is generated by an authentication service when the credentials are determined to be valid;
• selecting, by the request intake manager, a database server to store, in a job database that stores statuses of web scraping jobs while the web scraping jobs are being executed, a job description generated based on the first API request from a plurality of database servers identified as enabled in a database server status table, wherein the plurality of database servers operate independently of one another and each database server is configured to manage data storage to at least a portion of the job database;
• assigning, by the request intake manager, a job identifier to the job description, wherein the job identifier encodes an identifier for the selected database server;
• transmitting, by the request intake manager, the job identifier to the client computing device;
• and
• transmitting, by the request intake manager, the job identifier and the job description to the selected database server.

A system for determining which servers are available to process web scraping jobs, comprising:

• a job database that stores statuses of web scraping jobs while the web scraping jobs are being executed;
• a plurality of database servers operating independently of one another, wherein each of the plurality of database servers is configured to manage storage in a portion of the job database;
• a request intake manager comprising one or more processors and a memory storing instructions that, when executed by the one or more processors, configures the request intake manager to:
• obtain a first token indicating an identity of a client associated with credentials received with a first application programming interface (API) request from a client computing device, wherein the first API request specifies a target website to scrape and the first token is generated by an authentication service when the credentials are determined to be valid;
• determine, based on periodic health checks, whether each of a plurality of database servers are to be enabled or disabled in a database server status table;
• select a database server to store a job description generated based on the first API from the plurality of database servers identified as enabled in the database server status table;
• assign a job identifier to the job description, wherein the job identifier encodes an identifier for the selected database server;
• transmit the job identifier to the client computing device;
• and
• transmit the job identifier and the job description to the selected database server.