System, Method, and Computer Program Product for Dynamic Optimization of Complex Queries in Data Lakehouse Query Engines

A computer-implemented method, comprising:

• receiving, with at least one processor, data associated with operations of computational resources of a computer cluster;
• receiving, with at least one processor, data associated with a query to be carried out by the computer cluster during a time interval;
• providing, with at least one processor, an input to a cluster load machine learning model to generate an output of the cluster load machine learning model, wherein the output comprises a prediction of load on the computer cluster required to carry out operations during the time interval;
• determining, with at least one processor, to adjust at least one session parameter associated with the query based on an output of the cluster load machine learning model, the data associated with computational resources of the computer cluster, and the data associated with a query to be carried out by the computer cluster;
• and
• adjusting, with at least one processor, the at least one session parameter associated with the query based on determining to adjust the at least one session parameter associated with the query.

A computer program product, comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:

• receive data associated with operations of computational resources of a computer cluster;
• receive data associated with a query to be carried out by the computer cluster during a time interval;
• provide an input to a cluster load machine learning model to generate an output of the cluster load machine learning model, wherein the output comprises a prediction of load on the computer cluster required to carry out operations during the time interval;
• determine whether to adjust at least one session parameter associated with the query based on an output of the cluster load machine learning model, the data associated with computational resources of the computer cluster, and the data associated with a query to be carried out by the computer cluster;
• adjust the at least one session parameter associated with the query based on determining to adjust the at least one session parameter associated with the query;
• and
• forego adjusting the at least one session parameter associated with the query based on determining not to adjust the at least one session parameter associated with the query.