Platform for performing multi-stage data operations

A computer system for chaining data operations, comprising:

• one or more data processing apparatuses including one or more processors, memory, and storage devices storing instructions that, when executed, cause the one or more processors to perform operations comprising:
• receiving, from a client device, a definition of a multi-stage data operation that is expressed in a user-facing syntax;
• translating the user-facing syntax of the multi-stage data operation into a definition of the multi-stage data operation that is expressed in an intermediate syntax;
• parsing, by a data processing platform, the intermediate syntax of the multi-stage data operation, to identify a series of data operations included in the multi-stage data operation;
• coordinating, by the data processing platform, execution of the series of data operations included in the multi-stage data operation, by:
• for each data operation in the series of data operations:
• selecting, by the data processing platform, a respective operation handler that is configured to execute the data operation;
• generating, by the data processing platform, domain-specific syntax that is used by the selected operation handler for executing the data operation;
• and
• providing, by the data processing platform and to the selected operation handler, the domain-specific syntax for executing the data operation;
• for a first data operation and a second data operation in the series of data operations:
• in response to determining that neither the first data operation nor the second data operation depend on input from another data operation, (i) identifying a first output variable that indicates output of the first data operation, and (ii) identifying a second output variable that indicates output of the second data operation;
• mapping, by the data processing platform, (i) a first portion of a data cache to the first output variable for storing first operation results data that results from execution of the first data operation, and (ii) a second portion of the data cache to the second output variable for storing second operation results data that results from execution of the second data operation;
• and
• coordinating a concurrent execution of the first data operation and the second data operation;
• and
• for a third data operation in the series of data operations:
• in response to identifying a first input variable that indicates input for the third data operation and that corresponds to the first output variable of the first data operation, and identifying a second input variable that indicates input for the third data operation and that corresponds to the second output variable of the second data operation, waiting for availability of the first operation results data and the second operation results data;
• retrieving, by the data processing platform, (i) the first operation results data from the first portion of the data cache that is mapped to the first input variable, and (ii) the second operation results data from the second portion of the data cache that is mapped to the second input variable, and
• providing the retrieved first operation results data and the retrieved second operation results data to its operation handler;
• and
• after all data operations in the series of data operations have been successfully executed, providing a result of the multi-stage data operation to the client device.