Systems and methods for dynamic query optimization for distributed computing environments

A computer system for facilitating a data exchange, comprising:

• one or more processors;
• and
• a memory, wherein the memory stores one or more programs configured for execution by the one or more processors, and the one or more programs comprise instructions for:
• receiving a query for execution at a distributed computing environment with computing constraints;
• analyzing the query to determine complexity of the query, wherein the analysis comprises:
• generating an execution plan for the query;
• determining a total data volume to be processed;
• and
• identifying types and number of operations to be performed;
• determining required computational resources based on the analysis and on a mapping of data volume and operation complexity to platform-specific compute units of the distributed computing environment, wherein the determination comprises:
• estimating a first number of processing units needed based on the total data volume;
• estimating a second number of processing units needed based on the types and number of operations;
• and
• calculating a total number of processing units required by adding the first number of processing units to the second number of processing units;
• selecting an execution environment for the query, wherein the selection comprises:
• identifying existing execution environments capable of providing the required computational resources;
• upon a determination that existing execution environments capable of providing the required computational resources do not exist, provisioning a new execution environment and designating the new execution environment as a chosen execution environment;
• upon a determination that existing execution environments capable of providing the required computational resources do exist:
• determining current workload status of the existing execution environments based on real time workload information;
• and
• determining whether any of the existing execution environments has a current workload below a predefined workload threshold;
• upon a determination that at least one of the existing execution environments has a current workload below the predefined workload threshold, choosing one of the existing execution environments, having the lowest current workload, as the chosen execution environment;
• and
• upon a determination that none of the existing execution environments has a current workload below the predefined workload threshold, provisioning a new execution environment and designating the new execution environment as the chosen execution environment;
• submitting the query to the chosen execution environment;
• monitoring the query execution until completion;
• updating a query performance database with execution statistics;
• and
• repeating receiving the query, analyzing the query, determining the required computational resources, selecting the execution environment, submitting the query, monitoring the query execution, and updating the query performance database, for multiple queries in parallel, wherein each query's execution environment selection is based on real-time workload information and wherein the parallel processing of multiple queries includes dynamically adjusting execution environment selections based on changing workload conditions.