Method and system for optimizing resource requirements on a distributed processing platform

A computing device configured to implement an execution of a method for optimizing at least one resource requirement on a distributed processing platform, the computing device comprising:

• a processor;
• a memory;
• and
• a communication interface coupled to each of the processor and the memory,
• wherein the processor is configured to:
• receive, via the communication interface, at least one text file based on a user input;
• generate an output file based on the at least one text file, wherein the output file comprises an explain plan associated with the at least one text file;
• identify, using a trained model, at least one operator in the output file;
• apply at least one rule to the at least one operator, wherein the at least one rule for the at least one operator is retrieved from at least one rule database;
• scan the at least one rule that is applied to the at least one operator;
• recommend, using the trained model, at least one change in the at least one rule based on the scan of the at least one rule;
• and
• generate at least one updated output file based on the recommendation of the at least one change in the at least one rule,
• wherein the processor is further configured to execute the at least one updated output file by:
• analyzing, using the trained model, the at least one updated output file;
• identifying at least one configuration associated with the distributed processing platform based on the analysis of the at least one updated output file;
• recommending the at least one identified configuration for the execution of the at least one updated output file;
• and
• executing the at least one updated output file using the at least one identified configuration,
• wherein the recommending of the at least one identified configuration for the execution of the at least one updated output file is associated with a memory and a number of cores required to be allocated in master and worker nodes in the distributed processing platform for optimized performance with respect to excessive data shuffles, memory leakages, and misconfigurations, and
• wherein the processor is further configured to further train the trained model by using the at least one identified configuration.

A non-transitory computer readable storage medium storing instructions for optimizing at least one resource requirement on a distributed processing platform, the instructions comprising executable code which, when executed by a processor, causes the processor to:

• receive at least one text file based on a user input;
• generate an output file based on the at least one text file, wherein the output file comprises an explain plan associated with the at least one text file;
• identify, using a trained model, at least one operator in the output file;
• apply at least one rule to the at least one operator, wherein the at least one rule for the at least one operator is retrieved from at least one rule database;
• scan the at least one rule that is applied to the at least one operator;
• recommend, using the trained model, at least one change in the at least one rule based on the scan of the at least one rule;
• and
• generate at least one updated output file based on the recommendation of the at least one change in the at least one rule,
• wherein when executed by the processor, the executable code further causes the processor to execute the at least one updated output file by:
• analyzing, using the trained model, the at least one updated output file;
• identifying at least one configuration associated with the distributed processing platform based on the analysis of the at least one updated output file;
• recommending the at least one identified configuration for the execution of the at least one updated output file;
• and
• executing the at least one updated output file using the at least one identified configuration,
• wherein the recommending of the at least one identified configuration for the execution of the at least one updated output file is associated with a memory and a number of cores required to be allocated in master and worker nodes in the distributed processing platform for optimized performance with respect to excessive data shuffles, memory leakages, and misconfigurations, and
• wherein when executed by the processor, the executable code further causes the processor to further train the trained model by using the at least one identified configuration.