Intermediate sorting for data set projection builds
AssigneeAmazon Technologies, Inc.
InventorsSagar Mundra, Sameer Jain
Life of this patent
Estimating the term…
Abstract
An intermediate sorted version of a projection of a data set is used to create the projection of the data set. A copy of the data set can be accessed to store an intermediate sorted version and determine a number of portions to the projection of the data set across a number of data stores. Then, distributed execution to create the projection of the data set to be performed from different portions of the number of portions to store at respective ones of the data stores.
Claims17
A system, comprising:
- a plurality of computing devices, respectively comprising at least one processor and a memory, that implement a distributed data store;
- wherein the distributed data store is configured to:
- receive a request to create a projection of a data set stored across a plurality of data stores according to a different schema;
- create the projection of the data set, wherein to create the projection of the data set, the distributed data store is configured to:
- access a copy of the data set to sort data obtained from the copy of the data set to include in the projection according to the different schema;
- store the sorted data as an intermediate sorted version of the projection of the data set according to the different schema;
- and
- determine a number of portions to be used to store the projection of the data set across the plurality of data stores;
- and
- distribute execution across a plurality of different computing resources to create the projection of the data set from different portions of the number of portions to store at respective ones of the plurality of data stores;
- and
- provide one or more progress indicators of creating the projection of the data set based, at least in part, on the determined number of portions.
A method, comprising:
- creating a projection of a data set stored across a plurality of data stores according to a different schema, comprising:
- accessing a copy of the data set to store an intermediate sorted version of the projection of the data set according to the different schema;
- and
- determining a number of portions to be used to store the projection of the data set across the plurality of data stores;
- and
- distributing execution to create the projection of the data set from different portions of the number of portions to store at respective ones of the plurality of data stores;
- and
- providing one or more progress indicators of creating the projection of the data set based, at least in part, on the determined number of portions.
One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:
- receiving a request to create a projection of a data set stored across a plurality of data stores according to a different schema;
- creating a projection of the data set, wherein creating the projection of the data set the program instructions cause the one or more computing devices to implement:
- causing access to a copy of the data set to store an intermediate sorted version of the projection of the data set according to the different schema;
- and
- determining a number of portions to be used to store the projection of the data set across the plurality of data stores;
- and
- causing distributed execution to create the projection of the data set from different portions of the number of portions to store at respective ones of the plurality of data stores;
- and
- providing one or more progress indicators of creating the projection of the data set based, at least in part, on the determined number of portions.
Dependencies are inferred from the claim text.