Dynamically loading graph-based computations

A data processing system for processing data using compiled dataflow graphs stored in a data storage system, the system including:

• at least one processor;
• and
• one or more machine-readable hardware storage devices storing instructions that are executable by the at least one processor to perform operations including:
• receiving multiple units of work that each include one or more work elements;
• and
• processing a unit of work of the multiple units of work, the processing including:
• analyzing the unit of work to determine a characteristic of the unit of work;
• identifying a compiled dataflow graph from multiple compiled dataflow graphs that were compiled before the unit of work was received, for processing a unit of work having the determined characteristic of the unit of work;
• loading the identified compiled dataflow graph into a data processing program being executed by the data processing system, wherein the data processing program is being executed before receipt of the unit of work;
• and
• generating one or more output work elements from at least one work element in the unit of work using the identified compiled dataflow graph.

A data processing system for processing data using compiled dataflow graphs stored in a data storage system, the system including:

• means for receiving multiple units of work that each include one or more work elements;
• and
• means for processing a unit of work of the multiple units of work, the processing including:
• analyzing the unit of work to determine a characteristic of the first unit of work;
• identifying a compiled dataflow graph from multiple compiled dataflow graphs that were compiled before the unit of work was received, for processing a unit of work having the determined characteristic of the unit of work;
• loading the identified compiled dataflow graph into a data processing program being executed by the data processing system, wherein the data processing program is being executed before receipt of the unit of work;
• and
• generating one or more output work elements from at least one work element in the unit of work using the identified compiled dataflow graph.

A computer-readable medium storing a computer program for processing data in a data processing system using compiled dataflow graphs stored in a data storage system, the computer program including instructions for causing a computer to perform operations comprising:

• receiving multiple units of work that each include one or more work elements;
• and
• processing a unit of work of the multiple units of work, the processing including:
• analyzing the unit of work to determine a characteristic of the unit of work;
• identifying a compiled dataflow graph from multiple compiled dataflow graphs that were compiled before the unit of work was received, for processing a unit of work having the determined characteristic of the unit of work;
• loading the identified compiled dataflow graph into a data processing program being executed by the data processing system, wherein the data processing program is being executed before receipt of the unit of work;
• and
• generating one or more output work elements from at least one work element in the unit of work using the identified compiled dataflow graph.

A method for processing data in a data processing system, the method including:

• receiving multiple units of work that each include one or more work elements;
• and
• processing a unit of work of the multiple units of work, the processing including:
• analyzing the unit of work to determine a characteristic of the unit of work;
• identifying a compiled sub-program from multiple compiled sub-programs that were compiled before the unit of work was received, for processing a unit of work having the determined characteristic of the unit of work;
• loading one of the identified compiled sub-program into a data processing program being executed by the data processing system before receipt of the unit of work;
• and
• generating one or more output work elements from at least one work element in the unit of work using the identified compiled sub-program.

A method for processing data in a data processing system using compiled dataflow graphs stored in a data storage system, the method including:

• receiving multiple units of work that each include one or more work elements;
• and
• processing a unit of work of the multiple units of work, the processing including:
• analyzing the unit of work to determine a characteristic of the unit of work;
• identifying a compiled dataflow graph from multiple compiled dataflow graphs that were compiled before the unit of work was received, for processing a unit of work having the determined characteristic of the unit of work;
• and
• loading the compiled dataflow graph into the data processing system by:
• retrieving data structures representing a first dataflow graph from which the compiled dataflow graph is compiled and loading the retrieved data structures into a second dataflow graph by embedding the first dataflow graph within a containing component of the second dataflow graph that is configured to provide a first process to execute the data processing components in the first dataflow graph;
• wherein the containing component of the second dataflow graph is further configured to provide a first application programming interface that the first process uses to communicate with a second process that accesses a library for a first remote database;
• and
• generating one or more output work elements from at least one work element in the unit of work using the compiled dataflow graphs.

A method for processing data in a data processing system using compiled dataflow graphs stored in a data storage system, the method including:

• receiving multiple units of work that each include one or more work elements;
• and
• processing a unit of work of the multiple units of work, the processing including:
• analyzing the unit of work to determine a characteristic of the unit of work;
• identifying a compiled dataflow graph from multiple compiled dataflow graphs that were compiled before the unit of work was received, for processing a unit of work having the determined characteristic of the unit of work;
• loading one of the compiled dataflow graphs into the data processing system by retrieving data structures representing a first dataflow graph from which the compiled dataflow graph is compiled and loading the retrieved data structures into a second dataflow graph by:
• embedding the first dataflow graph within a containing component of the second dataflow graph, the embedding including:
• connecting a flow of input work elements received from a link connected to an input of the containing component to an input of the first dataflow graph;
• and
• connecting the generated one or more output work elements to a link connected to an output of the containing component;
• and
• generating one or more output work elements from at least one work element in the unit of work using the compiled dataflow graph.

A computer-readable medium storing a computer program for processing data in a data processing system using compiled sub-programs stored in a data storage system, the computer program including instructions for causing a computer to perform operations comprising:

• receiving multiple units of work that each include one or more work elements;
• and
• processing unit of work of the multiple units of work, the processing including:
• analyzing the unit of work to determine a characteristic of the unit of work;
• identifying a compiled sub-program from multiple compiled sub-programs that were compiled before the unit of work was received, for processing a unit of work having the determined characteristic of the unit of work;
• loading one of the identified compiled sub-program into a data processing program being executed by the data processing system before receipt of the unit of work;
• and
• generating one or more output work elements from at least one work element in the unit of work using the identified compiled sub-program.

A data processing system comprising:

• one or more processors;
• and
• a computer-readable medium storing instructions that are executable by the one or more processors perform operations comprising:
• receiving multiple units of work that each include one or more work elements;
• and
• processing unit of work of the multiple units of work, the processing including:
• analyzing the unit of work to determine a characteristic of the unit of work;
• identifying a compiled sub-program from multiple compiled sub-programs that were compiled before the unit of work was received, for processing a unit of work having the determined characteristic of the unit of work;
• loading one of the identified compiled sub-program into a data processing program being executed by the data processing system before receipt of the unit of work;
• and
• generating one or more output work elements from at least one work element in the unit of work using the identified compiled sub-program.