Resource estimation for queries in large-scale distributed database system

A method of managing queries on on-line advertisement data, comprising:

• receiving queries for a plurality of query clients from a corresponding plurality of query client devices, wherein the queries utilize a flexible protocol and upon execution vary significantly in complexity, data input size, cluster usage, and results, wherein execution of each received query is to be distributed and processed across a plurality of cluster nodes;
• generating an initial model having a plurality of weights for applying to each of a plurality of different feature vectors of a plurality of queries, which are received for a plurality of query clients via a plurality of query devices, so as to predict an estimation of resource usage for execution of each query, each query's corresponding feature vector having a plurality of values pertaining to the query and a system status, wherein each query's corresponding feature vector includes input data items from the query's content and database parameter items for specifying different types of queried data for execution of the query and that are not specified in the query's content and wherein the input data items pertain to at least the query's complexity, data input size, cluster usage, and results;
• prior to adjusting the weights of the initial model for a particular one of the query clients, tracking actual resource usage during execution of each of a first set of received queries for the particular query client;
• dynamically adjusting the weights of the initial model for the particular query client to generate an adjusted model for individual use with the particular query client's subsequently received queries, wherein the weights are only adjusted if a condition is met when comparing actual resource usage and estimation of resource usage for executing a current query for the particular query client;
• and
• providing the estimation of resource usage for each query to the corresponding query client device and thereafter, in response to receipt of the estimation by the corresponding query client device, receive input from such corresponding query client device and specifying whether to proceed with the corresponding query.

One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:

• receiving queries for a plurality of query clients from a corresponding plurality of query client devices, wherein the queries utilize a flexible protocol and upon execution vary significantly in complexity, data input size, cluster usage, and results, wherein execution of each received query is to be distributed and processed across a plurality of cluster nodes;
• generating an initial model having a plurality of weights for applying to each of a plurality of different feature vectors of a plurality of queries, which are received for a plurality of query clients via a plurality of query devices, so as to predict an estimation of resource usage for execution of each query, each query's corresponding feature vector having a plurality of values pertaining to the query and a system status, wherein each query's corresponding feature vector includes input data items from the query's content and database parameter items for specifying different types of queried data for execution of the query and that are not specified in the query's content and wherein the input data items pertain to at least the query's complexity, data input size, cluster usage, and results;
• prior to adjusting the weights of the initial model for a particular one of the query clients, tracking actual resource usage during execution of each of a first set of received queries for the particular query client;
• dynamically adjusting the weights of the initial model for the particular query client to generate an adjusted model for individual use with the particular query client's subsequently received queries, wherein the weights are only adjusted if a condition is met when comparing actual resource usage and estimation of resource usage for executing a current query for the particular query client;
• and
• providing the estimation of resource usage for each query to the corresponding query client device and thereafter, in response to receipt of the estimation by the corresponding query client device, receive input from such corresponding query client device and specifying whether to proceed with the corresponding query.