Continuous builds of derived datasets in response to other dataset updates

A computer system comprising:

• one or more processors;
• one or more non-transitory computer-readable storage media coupled to the one or more processors and storing one or more sequences of instructions which when executed cause the one or more processors to perform:
• storing, in a distributed data storage system, a plurality of raw datasets, the plurality of raw datasets comprising data drawn from one or more data sources;
• obtaining a definition of at least one derived dataset of a data pipeline;
• creating and storing a dependency graph in memory, the dependency graph reflecting the definition and representing the at least one derived dataset, the plurality of raw datasets, and a plurality of intermediate derived datasets on which the at least one derived dataset depends, wherein the dependency graph specifies that the at least one derived dataset depends on, at least, a stored first dataset of the plurality of raw datasets, and a stored second dataset of the plurality of raw datasets;
• reading run window configuration data associated with the first dataset and the second dataset, and specifying one or more time periods based on the run window configuration data;
• detecting a first update to the first dataset of the plurality of raw datasets, wherein the first update comprises a change to the stored first dataset upon which the at least one derived dataset depends;
• based on the dependency graph, initiating a first build of a first intermediate derived dataset that depends on the first dataset, the first build comprising at least applying one or more transformations to the first dataset that transforms data of the first dataset into data of the first intermediate derived dataset;
• initiating a second build of a second intermediate derived dataset that uses the first intermediate derived dataset and that is next in order in the data pipeline according to the dependency graph, the second build comprising at least applying one or more transformations to the first intermediate derived dataset that transforms data of the first intermediate derived dataset into data of the second intermediate derived dataset;
• after detecting the first update to the first dataset and prior to producing an updated version of the at least one derived dataset using the second dataset, detecting a second update to the second dataset from among the plurality of raw datasets or intermediate derived datasets on which the at least one derived dataset depends, wherein the second update comprises a change to the stored second dataset upon which the at least one derived dataset depends, the second dataset being independent from the first dataset in the dependency graph;
• based on determining that a then-current time is within one of the one or more time periods specified in the run window configuration data and based on the dependency graph, initiating a third build of a third intermediate derived dataset that depends on the second dataset, the third build comprising at least applying one or more transformations to the second dataset that transforms data of the second dataset into data of the third intermediate derived dataset;
• based on the dependency graph, initializing a fourth build that is next in the data pipeline using the third intermediate derived dataset and the second intermediate derived dataset to produce at least one updated version of the at least one derived dataset of the data pipeline, wherein the at least one derived dataset is produced using data from the third intermediate derived dataset and the second intermediate derived dataset;
• detecting and initiating continuously as other updates to other datasets are received.