Unit testing of components of dataflow graphs

A system implemented by data processing system for defining a unit test for a dataflow graph including a plurality of components, the system including:

• a data storage storing instructions;
• and
• at least one processor configured to execute the instructions stored by the data storage to perform operations including:
• accessing a dataflow graph including components and one or more dataflows representing flows of data records among the components, each of the components including executable logic including a set of instructions for processing the data records;
• receiving an indication of a portion of the dataflow graph, the portion including a subset of the components for being tested separate from another portion of the dataflow graph including other components;
• receiving parameters including:
• an output parameter indicative of expected one or more output data records to be generated by execution of the executable logic of the subset of the components of the portion of the dataflow graph in response to receiving one or more input data records, and
• a source parameter that identifies:
• a source of the one or more input data records, and
• one or more fields of the one or more input data records for being processed by the subset of the components during testing of the indicated portion of the dataflow graph, each of the one or more fields containing one or more values;
• and
• defining a unit test for the subset of the components, the unit test being defined based on the parameters, the unit test being configured to test the portion of the dataflow graph separate from the other portion of the dataflow graph by causing operations including:
• based on the source parameter that identifies the source of the one or more input data records, providing the one or more input data records to the indicated portion of the dataflow graph;
• causing processing of data of the one or more fields of the one or more input data records to generate one or more output data records, the processing including executing the executable logic including the set of the instructions for the subset of the components of the portion of the dataflow graph;
• based on the processing, generating results data indicating a correspondence between the generated one or more output data records and the expected one or more output data records indicated by the output parameter;
• and
• causing generation of structured data based on a combination of the results data, the one or more input data records, and the dataflow graph, wherein the structured data are stored in a data storage associated with a hash value for version control between a first version of the unit test defined by the parameter set and a second version of the unit test defined by a different parameter set of a different version of the structured data,
• the hash value for version control representing changes among prior versions of the structured data, subsequent versions of the structured data, or both relative to the structured data for version control between the first version of the unit test from the second version of the unit test.

One or more non-transitory computer readable media storing instructions for defining a unit test for a dataflow graph including a plurality of components, the instructions configured to cause at least one processor to perform operations including:

• accessing a dataflow graph including components and one or more dataflows representing flows of data records among the components, each of the components including executable logic including a set of instructions for processing the data records;
• receiving an indication of a portion of the dataflow graph, the portion including a subset of the components for being tested separate from another portion of the dataflow graph including other components;
• receiving parameters including:
• an output parameter indicative of expected one or more output data records to be generated by execution of the executable logic of the subset of the components of the portion of the dataflow graph in response to receiving one or more input data records, and
• a source parameter that identifies:
• a source of the one or more input data records, and
• one or more fields of the one or more input data records for being processed by the subset of the components during testing of the indicated portion of the dataflow graph, each of the one or more fields containing one or more values;
• and
• defining a unit test for the subset of the components, the unit test being defined based on the parameters, the unit test being configured to test the portion of the dataflow graph separate from the other portion of the dataflow graph by causing operations including:
• based on the source parameter that identifies the source of the one or more input data records, providing the one or more input data records to the indicated portion of the dataflow graph;
• causing processing of data of the one or more fields of the one or more input data records to generate one or more output data records, the processing including executing the executable logic including the set of the instructions for the subset of the components of the portion of the dataflow graph;
• based on the processing, generating results data indicating a correspondence between the generated one or more output data records and the expected one or more output data records indicated by the output parameter;
• and
• causing generation of structured data based on a combination of the results data, the one or more input data records, and the dataflow graph, wherein the structured data are stored in a data storage associated with a hash value for version control between a first version of the unit test defined by the parameter set and a second version of the unit test defined by a different parameter set of a different version of the structured data,
• the relationship hash value for version control representing changes among prior versions of the structured data, subsequent versions of the structured data, or both relative to the structured data for version control between the first version of the unit test from the second version of the unit test.