Translation techniques for distributed datasets

A method comprising:

• receiving first code in a first programming language, the first code including a reference to a first dataset, the first dataset comprises a resilient distributed dataset (RDD), the RDD comprises information defining a set of partitions and including a set of data to be partitioned into the set of partitions, and the RDD is in a first format that is incompatible with a database system;
• generating a directed acyclic graph (DAG) of the RDD;
• serializing the DAG;
• generating a database table based on the serialized DAG, the database table comprising a column corresponding to an object including the serialized DAG, a first column corresponding to partition identifier, and a second column corresponding to a serialized partition object;
• determining that the first code includes an action command that triggers an output of results or storing the results, wherein converting the first code to second code is in response to the determining that the first code includes the action command:
• in response to determining that the first code includes the action command, converting the first code to:
• second code in a second programming language different than the first programming language, and
• at least one database query in a second format different than the first programming language;
• and
• sending the converted first code and the at least one database query to a database system for execution, the converted first code being compatible with the database system.

A non-transitory computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:

• receiving first code in a first programming language, the first code including a reference to a first dataset, the first dataset comprises a resilient distributed dataset (RDD), the RDD comprises information defining a set of partitions and including a set of data to be partitioned into the set of partitions, and the RDD is in a first format that is incompatible with a database system;
• generating a directed acyclic graph (DAG) of the RDD;
• serializing the DAG;
• generating a database table based on the serialized DAG, the database table comprising a column corresponding to an object including the serialized DAG, a first column corresponding to partition identifier, and a second column corresponding to a serialized partition object;
• determining that the first code includes an action command that triggers an output of results or storing the results;
• in response to determining that the first code includes the action command, converting the first code to:
• second code in a second programming language different than the first programming language, and
• at least one database query in a second format different than the first programming language;
• and
• sending the converted first code and the at least one database query to a database system for execution, the converted first code being compatible with the database system.