Apparatuses and methods for programming parallel computers

A publicly available computer system providing an environment for parallel programming including:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream of records, or an output port including means for receiving a data stream of records from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• a publicly defined interface for enabling users to define a data flow graph, including means for enabling the users to give:
• instructions to select a specific one of said operators for inclusion in said graph
• instructions to select a specific data object, which is capable of supplying or receiving a data stream of one or more records, for inclusion in said graph, and
• instructions to associate a data link with a specific communication port of an operator in the graph, which data link defines a path for the communication of a data stream of one or more records between its associated communications port and either a specific data object or the specific communication port of another specific operator in said graph;
• means for automatically parallelizing the execution of a data flow graph equivalent to that defined by the users, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators selected by said instructions for inclusion in the graph by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input.

A computer system comprising:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream, or an output port including means for receiving a data stream from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• means for enabling a user to define a specific data flow graph including at least two of said operators, said means including:
• means for enabling the user to explicitly instruct said system to individually select each one or more of said operators in said graph;
• means for enabling the user to explicitly instruct said system to individually select each of one or more data objects for inclusion in said graph, each of which is capable of supplying or receiving a data stream;
• means for enabling a user to explicitly instruct said system to individually connect a specific data link between a given communication port in said graph and either a specific communication port of another operator in said graph selected by the user in said instruction or a specific user selected data object selected by the user in said instruction;
• means for automatically parallelizing the execution of a data flow graph equivalent to that defined by the user, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators selected by the user for inclusion in the graph by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input.

A process of executing a computer program comprising performing the following steps during the execution of said program:

• storing each of a plurality of modular parallelizable operators, each of which defines:
• operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port for providing the operation programming a data stream, or an output port for receiving a data stream from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• storing a specific data flow graph which has been defined prior to the execution of said program, said graph including:
• individual identification of each of two or more of said operators which have been selected for inclusion in said graph;
• and
• definition of each of a plurality of data links each associated with an individual communication port of an operator in said graph, each of which data link definitions identifies a specific communication port of a specific operator at one end and, at the other end, identifies either a specific communication port of another specific operator or a data object, each of which is capable of supplying or receiving a data stream to or from said data link;
• automatically parallelizing the execution of a data flow graph equivalent to the stored pre-defined graph, said graph parallelizing step including:
• automatically parallelizing the execution of individual operators in said equivalent graph by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator;
• and
• automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input.

A computer system comprising:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream of records, or an output port including means for receiving a data stream of records from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• means for defining a specific data flow graph including:
• one or more of said operators
• a data link associated with the communication ports of the operators in said graph, each of which indicates a data stream of one or more records from the output port of one operator to the input port of another, from a data-supplying object to the input port of an operator, or from the output port of an operator to a data-receiving object;
• means for automatically parallelizing the execution of a data flow graph equivalent to said specific graph, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators in said equivalent graph by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors, with each instance of a given operator having a corresponding input and output port for each input and output port of said given operator;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input, and
• means for checking the specific graph or equivalent graph for one or more indications that the graph parallelizing means may not be able to successfully parallelize the execution of the specific graph and, for producing an action as a function of whether or not such indications of possible difficulties are detected.

A computer system comprising:

• means defining a data flow graph, including:
• one or more of operators each defining operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream, or an output port including means for receiving a data stream from the operation programming, and
• a data link associated with each of individual communication ports, each of which indicates a data stream from the output port of one operator to the input port of another, from a data-supplying object to the input of an operator, or from the output of an operator to a data-receiving object;
• wherein at least one of the operators is a program-execution operator having:
• operation programming for issuing a command to execute an instance of a separate computer program, and
• means for communicating through said program-execution operator's communication port between the instance of the separate program and a data stream indicated by said communication port's associated data link;
• and
• means for automatically parallelizing the execution of the data flow graph, said graph parallelizing means including:
• means for automatically parallelizing the execution of the graph by causing a separate instance of each of one or more operators in the graph, including said program-execution operator, to be run on each of multiple processors, with each instance of a given operator separately executing its operation programming and with each such instance of a given operator having a corresponding input or output port for each input or output port of the given operator;
• and
• means for automatically partitioning the data stream indicated by the data link corresponding to each input of the instances of a given operator between the inputs of the individual instances of the given operator.

A computer system including:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream, or an output port including means for receiving a data stream from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• means for defining a data flow graph including:
• one or more of said operators for inclusion in said graph, and
• a data link associated with the communication ports of the operators in said graph, each of which indicates a data stream from the output port of one operator to the input port of another, from a data-supplying object to the input of an operator, or from the output of an operator to a data-receiving object;
• means for automatically parallelizing the execution of a data flow graph equivalent to that defined by the user, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators selected by the user for inclusion in the graph by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the given operator's partitioning indication for the corresponding input, wherein at least one of said modular parallelizable operators is a composite operator which defines:
• a data flow sub-graph including:
• two or more others of said operators;
• one or more data links each connecting an output port and an input port of different operators in the sub-graph;
• and
• one or more external communication ports, either an input or output port, for linking to data links external to said composite operator, each of which external ports includes associated means for connecting a data stream connected to said external port to a communication port of one of the operators of said sub-graph;
• said means for automatically parallelizing the execution of individual operators includes means for automatically parallelizing each parallelizable operator in said subgraph;
• and
• said means for automatically partitioning includes means for partitioning the data stream supplied to corresponding inputs of the instances of each given parallelized operator of the sub-graph as a function of the given operator's partitioning method indication for the corresponding input.

A computer system comprising:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream of one or more records, or an output port including means for receiving a data stream of one or more records from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• means for defining a data flow graph including:
• one or more of said operators
• a data link associated with the communication ports of the operators in said graph, each of which indicates a data stream of one or more records from the output port of one operator to the input port of another, from a data-supplying object to the input of an operator, or from the output of an operator to a data-receiving object;
• means for automatically parallelizing the execution of a data flow graph equivalent to that accessed, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators in said equivalent graph by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input,
• wherein at least one of the parallelizable operators is an analytical apply operator having:
• an input port;
• modeling values which have been trained from the values of multiple different fields in each of a plurality of records;
• and
• operation programming for performing an operation based on the values of multiple different fields in an individual record supplied to it as a function of said modeling values.

A computer system comprising:

• a plurality of modular operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream of one or more records, or an output port including means for receiving a data stream of one or more records from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• an indication for each operator of whether or not it is intended to be parallelizable;
• means for defining a data flow graph including:
• one or more of said operators
• a data link associated with the communication ports of the operators in said graph, each of which indicates a data stream of one or more records from the output port of one operator to the input port of another, from a data-supplying object to the input of an operator, or from the output of an operator to a data-receiving object;
• means for automatically parallelizing the execution of a data flow graph equivalent to that accessed, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators in said equivalent graph which have an indication that they are intended to be parallelizable by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator and for causing a single instance of operators which an indication that they are not intended to be parallelized to be run on one processor;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method
• indication for the given operator's corresponding input and the number of processors instances of the operator are being run on;
• and
• wherein at least one of the operator is an analytical build operator having an input port and operator programming for building an analytical model of records supplied to said input port, which model, when built, includes modeling values trained from the values of multiple fields in each of a plurality of records supplied to the operator's input port.

A computer system comprising:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream of one or more records, or an output port including means for receiving a data stream of one or more records from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• means for defining a data flow graph including:
• one or more of said operators
• a data link associated with the communication ports of the operators in said graph, each of which indicates a data stream of one or more records from the output port of one operator to the input port of another, from a data-supplying object to the input of an operator, or from the output of an operator to a data-receiving, object;
• means for automatically parallelizing the execution of a data flow graph equivalent to that accessed, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators in said equivalent graph by causing a separate instance of each such operator, including its associated operation programming, to he run on each of multiple processors, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input, and
• wherein at least one parallelizable operators includes an operator which includes operation programming for performing I/O with a persistent data object other than through a communication port of said operator.

A computer system comprising:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream of one or more records, or an output port including means for receiving a data stream of one or more records from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• means for defining a data flow graph including:
• one or more of said operators
• a data link associated with the communication ports of the operators in said graph, each of which indicates a data stream of one or more records from the output port of one operator to the input port of another, from a data-supplying object to the input of an operator, or from the output of an operator to a data-receiving object;
• means for automatically parallelizing the execution of a data flow graph equivalent to that accessed, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators in said equivalent graph by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors under a given operating system, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input,
• wherein at least one of said parallelizable operators includes operation programming for performing I/O with inter-process communication mechanisms provided by said given operating system, so that parallel instances of said operator perform such I/O in parallel.

A computer system comprising:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream of one or more records, or an output port including means for receiving a data stream of one or more records from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• means for defining a data flow graph including:
• one or more of said operators
• a data link associated with the communication ports of the operators in said graph, each of which indicates a data stream of one or more records from the output port of one operator to the input port of another, from a data-supplying object to the input of an operator, or from the output of an operator to a data-receiving object;
• means for automatically parallelizing the execution of a data flow graph equivalent to that accessed, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators in said equivalent graph by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors under a given operating system, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input,
• wherein at least one of said parallelizable operators includes operation programming for performing i/o with a user console, so that when parallelized each instance of said operator will separately communicate with a user console.

A computer system comprising:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream of one or more records, or an output port including means for receiving a data stream of one or more records from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• means for defining a data flow graph including:
• one or more of said operators
• a data link associated with the communication ports of the operators in said graph, each of which indicates a data stream of one or more records from the output port of one operator to the input port of another, from a data-supplying object to the input of an operator, or from the output of an operator to a data-receiving object;
• means for automatically parallelizing the execution of a data flow graph equivalent to that accessed, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators in said equivalent graph by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors, with each instance of a given operator having a corresponding input port and an output port for each input and output port of said given operator;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input, and
• wherein:
• at least a one of said parallelizable operators is a two-output operator which includes at least two output ports and operation programming for supplying a data stream of one or more records to each of said two output ports;
• the means for defining a graph includes means for defining a given graph including a down stream fork comprised of at least one of said two-output operators, in which each of said two-output operator's two output ports is connected by a data link to either the input port of another operator or a data-receiving object;
• and
• the means for parallelizing includes means for automatically parallelizing said graph including said downstream fork.

A computer system comprising:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• operation programming for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operation programming a data stream of one or more records, or an output port including means for receiving a data stream of one or more records from the operation programming, and
• an indication for each of the operator's input ports, if any, of a partitioning method to be applied to the data stream supplied to the input port;
• means for defining a data flow graph including:
• one or more of said operators
• a data link associated with the communication ports of the operators in said graph, each of which indicates a data stream of one or more records from the output port of one operator to the input port of another, from a data-supplying object to the input of an operator, or from the output of an operator to a data-receiving object;
• means for automatically parallelizing the execution of a data flow graph equivalent to that accessed, said graph parallelizing means including:
• means for automatically parallelizing the execution of individual operators in said equivalent graph by causing a separate instance of each such operator, including its associated operation programming, to be run on each of multiple processors under a given operating system, with each instance of a given operator having a corresponding input and output port for each input and output port of the given operator;
• and
• means for automatically partitioning the data stream supplied to the corresponding inputs of the instances of a given operator as a function of the partitioning method indication for the given operator's corresponding input,
• wherein at least one of said parallelizable operators includes a field transfer definition, existing independently of any graph in which that operator occurs, specifying which fields of a records supplied to an input port of said operator will have their value copied un-changed to a corresponding record supplied by the operator to an output port of the operator.

A computer system comprising:

• a plurality of modular parallelizable operators stored in a computer readable memory, each of which defines:
• an operator routine which can be called for performing an operation;
• and
• one or more communication ports, each of which is either an input port including means for providing the operator routine a data stream of one or more records, or an output port including means for receiving a data stream of one or more records from the operator routine, and
• means for defining a given data flow graph including:
• one or more of said operators;
• a data link associated with the communication ports of the operators in said graph, each of which indicates a data stream of one or more records from the output port of one operator to the input port of another, from a data-supplying object to the input of an operator, or from the output of an operator to a data-receiving object;
• means for automatically simulating parallel execution of a data flow graph equivalent to said given graph, said graph simulation means including:
• means for automatically simulating parallel execution of each of one or more individual operators in said equivalent graph by making a plurality of separate sequential calls to the operator routine of each such operator, with each such call having an associated input and output port for each input and output port of the call's associated individual operator, enabling each such call to the operator routine to receive and output one or more data streams, respectively, if its associated operator has one or more input or output ports;
• and
• means for automatically buffering a data stream produced at an output port associated with an individual call to an operator routine where the operator routine's associated operator has an output port connected by a data link to the input port of another operator in said graph;
• means for automatically supplying a buffered data stream produced at the output port associated with one of multiple calls to the operator routine of a first operator to the input port associated with one of multiple calls to the operator routine of a second operator where the output port of the first operator is connected to the input port of the second operator in said equivalent graph.