Parallel processing of data for an untrusted application

A computer-implemented method comprising:

• receiving, by an untrusted worker process executing in an untrusted worker environment of a system from one or more trusted processes that manage data access to the untrusted worker environment, a revised dataflow graph comprising a) deferred, combined parallel operations that are associated with one or more untrusted functions of an untrusted application received by the system and b) deferred parallel data objects corresponding to a data parallel pipeline that was included in the untrusted application;
• executing, by the untrusted worker process, the untrusted functions associated with the deferred, combined parallel operations to produce materialized parallel data objects that correspond to the deferred parallel data objects;
• and
• providing, by the untrusted worker process, the materialized parallel data objects to the one or more trusted processes.

A non-transitory computer-readable medium storing instructions operable when executed to cause at least one processor to perform operations comprising:

• receiving, by an untrusted worker process executing in an untrusted worker environment of a system from one or more trusted processes that manage data access to the untrusted worker environment, a revised dataflow graph comprising a) deferred, combined parallel operations that are associated with one or more untrusted functions of an untrusted application received by the system and b) deferred parallel data objects corresponding to a data parallel pipeline that was included in the untrusted application;
• executing, by the untrusted worker process, the untrusted functions associated with the deferred, combined parallel operations to produce materialized parallel data objects that correspond to the deferred parallel data objects;
• and
• providing, by the untrusted worker process, the materialized parallel data objects to the one or more trusted processes.

A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• receiving, by an untrusted worker process executing in an untrusted worker environment of a system from one or more trusted processes that manage data access to the untrusted worker environment, a revised dataflow graph comprising a) deferred, combined parallel operations that are associated with one or more untrusted functions of an untrusted application received by the system and b) deferred parallel data objects corresponding to a data parallel pipeline that was included in the untrusted application;
• executing, by the untrusted worker process, the untrusted functions associated with the deferred, combined parallel operations to produce materialized parallel data objects that correspond to the deferred parallel data objects;
• and
• providing, by the untrusted worker process, the materialized parallel data objects to the one or more trusted processes.