Systems and Methods for Efficient Data Preprocessing of Machine Learning Workloads

A system for pre-processing a set of data for use in training a model or for use as an input to a trained model, comprising:

• one or more electronic processors configured to execute a set of computer-executable instructions;
• and
• one or more non-transitory computer-readable media containing the set of computer-executable instructions, wherein when executed, the instructions cause the one or more electronic processors or an apparatus or device in which they are contained to
• represent the pre-processing of a dataset as a sequence of one or more data conversion and data transformation operations;
• represent the sequence of one or more data conversions and data transformations and associated dependencies as a directed acyclic graph (DAG), wherein the nodes of the directed acyclic graph represent a state of the processing of the set of data, and an edge connecting two nodes represents a type of processor used to perform a data conversion or transformation;
• separate the directed acyclic graph into a set of sub-graphs, wherein each sub-graph represents one or more operations executed by a specific type or class of processor;
• rebalance one or more partitions in the data input to, or output by an operation to have substantially the same size after each operation executed by a specific type or class of processor;
• associate each sub-graph with an execution time and memory used to process the data input to a sub-graph using the one or more operations represented by the sub-graph;
• and
• for each sub-graph, execute the one or operations associated with a sub-graph with the rebalanced partitions of the datasets using the specific type of or class of processor.

One or more non-transitory computer-readable media comprising a set of computer-executable instructions that when executed by one or more programmed electronic processors, cause the processors or an apparatus or device in which they are contained to:

• represent the pre-processing of a dataset as a sequence of one or more data conversion and data transformation operations;
• represent the sequence of one or more data conversions and data transformations and associated dependencies as a directed acyclic graph (DAG), wherein the nodes of the directed acyclic graph represent a state of the processing of the set of data, and an edge connecting two nodes represents a type of processor used to perform a data conversion or transformation;
• separate the directed acyclic graph into a set of sub-graphs, wherein each sub-graph represents one or more operations executed by a specific type or class of processor;
• rebalance one or more partitions in the data input to, or output by an operation to have substantially the same size after each operation executed by a specific type or class of processor;
• associate each sub-graph with an execution time and memory used to process the data input to a sub-graph using the one or more operations represented by the sub-graph;
• and
• for each sub-graph, execute the one or operations associated with a sub-graph with the rebalanced partitions of the datasets using the specific type of or class of processor.