Compiled workflow code for distributed compute engine

A non-transitory computer readable medium storing computer-executable instructions that, when executed, cause a computer system to perform operations comprising:

• storing a library of computer-executable code for interfacing with a distributed compute engine, wherein the computer-executable code comprises functions that are configured to be executed by source code provided by a user device;
• receiving workflow source code for a workflow from a user device, wherein the source code comprises a primary set of instructions that define the workflow from a user and an interfacing set of instructions that cause the execution of a subset of the computer-executable code in the library;
• identifying the subset of the computer-executable code in the reference library corresponding to the interfacing set of instructions;
• generating compiled workflow code based on the received workflow source code and the identified subset of computer-executable code, wherein the compiled workflow code comprises machine code for the workflow source code and machine code for the identified subset of computer-executable code;
• initializing a plurality of executors within the distributed compute engine based on the compiled workflow code;
• accessing input data for inputting to the workflow;
• selecting, for one of the executors, a portion of the compiled workflow code for the executor to execute;
• and
• transmitting instructions to the executor to execute the selected portion of the compiled workflow code on a subset of the accessed input data.