Distributed dataset modification, retention, and replication

One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for managing data in a distributed dataset stored in partitions, the operations comprising:

• dividing the partitions into a plurality of generations;
• accumulating events that define changes to be applied to data of the distributed dataset;
• and
• in response to a triggering event occurring for a first generation of the plurality of generations, rolling a first partition out of the first generation, the rolling comprising:
• cleaning data of the first partition using a scrubbing process based on the defined changes to be applied to the data in the first partition,
• creating a second partition and adding the second partition to a second generation of the plurality of generations of the dataset,
• writing the cleaned data of the first partition to the second partition in the second generation, and
• marking the first partition for deletion in the first generation.

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for managing data in a distributed dataset stored in partitions, the operations comprising:

• dividing the partitions into a plurality of generations;
• accumulating events that define changes to be applied to data of the distributed dataset;
• and
• in response to a triggering event occurring for a first generation of the plurality of generations, rolling a first partition out of the first generation, the rolling comprising:
• cleaning data of the first partition using a scrubbing process based on the defined changes to be applied to the data in the first partition,
• creating a second partition and adding the second partition to a second generation of the plurality of generations of the dataset,
• writing the cleaned data of the first partition to the second partition in the second generation, and
• marking the first partition for deletion in the first generation.