US 2023/0409402 A1 · METHOD AND SYSTEM FOR PREDICTING OPTIMAL CONFIGURATION PARAMETERS FOR A DATA ANALYTICS ENGINE

Abstract

The invention relates to a method and system for predicting optimal configuration parameters for a data analytics engine. The method and system utilizes Supervised Machine Learning (ML) techniques for predicting optimal configuration parameters for the data analytics engine to run a specific application in a given time frame, wherein the optimal configuration includes estimating an optimal number of executor instances in real-time. The invention analyzes input parameters such as data metrics, software metrics, and hardware metrics to determine if the input parameters comprise a new dataset or a continuous dataset and deploy one or more models such as a Forecasting Model and a Regression Model. The one or more models then derive and allocate the possible number of executor instances. The executor instances allocated can be a number of executors, executor cores, the memory of executors, driver memory, and parallelism.

Claims14

Dependent claims

A computer-implemented method for predicting optimal configuration parameters for a data analytics engine, the method comprising:

receiving, by one or more processors, input parameters, wherein the input parameters comprise data metrics, software metrics, and hardware metrics;
analyzing, by the one or more processors, metadata of the input parameters, wherein the analysis comprises determining if the input parameters comprise a new dataset or a continuous dataset;
deploying, by the one or more processors, one or more models from a plurality of models based on the metadata of the input parameters, wherein the plurality of models comprise at least one of a Forecasting Model, and a Regression Model;
deriving, by the one or more processors using the one or more models, an optimum number of executor instances for a specific type of application in a given time frame;
and
allocating, by the one or more processors, the optimum number of executor instances for building optimal configuration parameters for the data analytics engine for the specific type of application in the given time frame.

3 more elements

A system comprising:

a memory;
a processor communicatively coupled to the memory, the processor configured to:
receive input parameters, wherein the input parameters comprise data metrics, software metrics, and hardware metrics;
analyze metadata of the input parameters, wherein the analysis comprises determining if the input parameters comprise a new dataset or a continuous dataset;
deploy one or models from a plurality of models based on metadata of the input parameters, wherein the plurality of models comprise at least one of a Forecasting Model, and a Regression Model;
derive, using the deployed one or more models, an optimum number of executor instances for a specific type of application in a given time frame;
and
allocate the optimum number of executor instances for building optimal configuration parameters for the data analytics engine for the specific type of application in the given time frame.

5 more elements

Dependencies are inferred from the claim text.

Method and system for predicting optimal configuration parameters for a data analytics engine

Life of this patent

Method and system for predicting optimal configuration parameters for a data analytics engine

Life of this patent

Abstract

Claims14