Distributed data storage using materialized intermediate partitions

A system, comprising:

• a processor;
• a communication interface;
• and
• a memory storing instructions that, when executed by the processor, cause the processor to:
• bucket a first dataset in a database into a partition, the first dataset including a first value and a first iteration of a second value, wherein the second value is a feature of the first value;
• bucket a second dataset in the database into the partition, the second dataset including the first value and a second iteration of the second value, wherein the second iteration of the second value is different than the first iteration of the second value;
• perform an UPSERT operation that updates the partition to include the first value and the second iteration of the second value;
• and
• transmit, via a communication interface, the updated partition to an external storage.

A computer-readable medium storing instructions that, when executed by a processor, cause the processor to:

• bucket a first dataset in a database into a partition, the first dataset including a first value and a first iteration of a second value, wherein the second value is a feature of the first value;
• bucket a second dataset in the database into the partition, the second dataset including the first value and a second iteration of the second value, wherein the second iteration of the second value is different than the first iteration of the second value;
• perform an UPSERT operation that updates the partition to include the first value and the second iteration of the second value;
• receive a query request, the query request including a request to return data and a merge key;
• bucket the updated partition by the merge key;
• and
• transmit, via a communication interface, a portion of the updated partition related to the received query request.