Data processing with integrated metadata generation and storage

A system for processing data based on a process definition that defines a sequence of processing steps, the system comprising:

• a metadata data store including metadata sets that correspond to the process definition;
• one or more processors;
• and
• computer-readable memory encoded with instructions that, when executed by the one or more processors, cause the system to:
• access the process definition of the system;
• execute an instance of a first processing step of the sequence of processing steps according to the process definition;
• generate first runtime metadata during an execution of the instance of the first processing step, the first runtime metadata including:
• metadata that corresponds to the instance of the first processing step and is correlated with a first set of metadata of the metadata sets that corresponds to each processing step of the sequence of processing steps, the metadata that corresponds to the instance of the first processing step including:
• an identification of the instance of the first processing step;
• a date and/or time associated with the instance of the first processing step;
• and
• a duration associated with the instance of the first processing step;
• and
• metadata that corresponds to an instance of a first data object that is produced or consumed by the first processing step;
• and
• store the first runtime metadata in the metadata data store during the execution of the instance of the first processing step such that the metadata data store integrates the first runtime metadata with the metadata sets that correspond to the process definition.