Method and system for acceleration of slower data processing codes in machine learning pipelines

A system comprising:

• an input/output interface to receive an unaccelerated data processing code comprising one or more statement of operations;
• a memory in communication with the one or more hardware processors, wherein the one or more hardware processors are configured to execute programmed instructions stored in the memory to:
• pre-process the received unaccelerated data processing code based on a predefined format to get a standardized code;
• identify at least one statement of the one or more statements of the standardized code having at least one operation is to be performed on a data frame, wherein the operation comprising an operator, and one or more operands, wherein the unaccelerated data processing code is processed to detect presence of the at least one operation, and if more than one operation is detected then the statement of the unaccelerated data processing code is broken into multiple statements of code to get the standardized code;
• generate a data processing representation and an ordered list of one or more data frame columns, using the standardized code, wherein each entry in the data processing representation holds information for a code statement and the code statement includes code with operations performed on a data frame, which needs to be accelerated, and wherein when the operations over data frame are absent in the code statement, then the code statement is ignored;
• identify an ordered list of one or more data frame columns using the generated data processing representation in the standardized code, wherein each of the one or more data objects present in the unaccelerated data processing code is identified by parsing the one or more data objects present in the unaccelerated data processing code line by line and checking the parsed objects and the data objects which are of type data frame columns are added in the ordered list;
• determine a filtering dictionary code from the identified ordered list of one or more data frame columns;
• and
• recommend an accelerated performant code from the data processing representation of the standardized code and the determined filtering dictionary code, wherein statements of the unaccelerated data processing code are visited in order and corresponding entry in the data processing representation are accessed and when statement in the unaccelerated data processing code does not have a corresponding data processing representation entry, the corresponding statement is put as it is in the accelerated code and when statement in the unaccelerated data processing code has a corresponding data processing representation entry, the information presented in corresponding data processing representation node is converted to a code by using list and NumPy based operation and the converted code is made part of the accelerated code, wherein the data processing representation node contains operation, operands, input data frame, and output data object.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• receiving, via an input/output interface, an unaccelerated data processing code comprising one or more statement of operations;
• pre-processing the received unaccelerated data processing code based on a predefined format to get a standardized code, wherein the standardized code comprising one operation to perform on a data frame;
• identifying at least one statement of the one or more statements of the standardized code having at least one operation that to be performed on the data frame, wherein the unaccelerated data processing code is processed to detect presence of the at least one operation, and if more than one operation is detected then the statement of the unaccelerated data processing code is broken into multiple statements of code to get the standardized code;
• generating a data processing representation and an ordered list of one or more data frame columns, using the standardized code, wherein each entry in the data processing representation holds information for a code statement and the code statement includes code with operations performed on a data frame, which needs to be accelerated, and wherein when the operations over data frame are absent in the code statement, then the code statement is ignored;
• identifying an ordered list of one or more data frame columns using the generated data processing representation in the standardized code, wherein each of the one or more data objects present in the unaccelerated data processing code is identified by parsing the one or more data objects present in the unaccelerated data processing code line by line and checking the parsed objects and the data objects which are of type data frame columns are added in the ordered list
• determining, a filtering dictionary code from the identified ordered list of one or more data frame columns;
• and
• recommending, an accelerated performant code from the data processing representation of the standardized code and the determined filtering dictionary code, wherein statements of the unaccelerated data processing code are visited in order and corresponding entry in the data processing representation are accessed and when statement in the unaccelerated data processing code does not have a corresponding data processing representation entry, the corresponding statement is put as it is in the accelerated code and when statement in the unaccelerated data processing code has a corresponding data processing representation entry, the information presented in corresponding data processing representation node is converted to a code by using list and NumPy based operation and the converted code is made part of the accelerated code, wherein the data processing representation node contains operation, operands, input data frame, and output data object.