Persistent Shuffle System
AssigneeGoogle Inc.
InventorsAlexander Gourkov Balikov, Marian Dvorsky, Yonggang Zhao
Life of this patent
Estimating the term…
Abstract
A method includes receiving a request to perform a shuffle operation on a data stream; receiving at least a portion of the data stream including a plurality of records, each including a key; storing each of the plurality of records in a persistent storage location assigned to a key range corresponding to keys included in the plurality of records; receiving a request from a consumer for a subset of the plurality of records including a range of keys; and upon receiving the request from the consumer, providing the subset of the plurality of records including the range of keys from the one or more persistent storage locations.
Claims20
A computer-implemented method comprising:
- receiving a request to perform a shuffle operation on a data stream that comprises a plurality of records that each include a key, the shuffle operation being an operation that groups the plurality of records in the data stream by key;
- producing an initial shuffler configuration, comprising assigning initial key ranges, that correspond to the data stream, to a first number of shufflers;
- determining, based on performance of the first number of shufflers, to reconfigure the initial shuffler configuration;
- and
- in response to determining to reconfigure the initial shuffler configuration, reconfiguring the initial shuffler configuration using a second number of shufflers, wherein the second number is different than the first number.
A system comprising:
- one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
- receiving a request to perform a shuffle operation on a data stream that comprises a plurality of records that each include a key, the shuffle operation being an operation that groups the plurality of records in the data stream by key;
- producing an initial shuffler configuration, comprising assigning initial key ranges that correspond to the data stream to a first number of shufflers;
- determining, based on performance of the first number of shufflers, to reconfigure the initial shuffler configurations;
- and
- in response to determining to reconfigure the initial shuffler configuration, reconfiguring the initial shuffler configuration using a second number of shufflers, wherein the first number is different than the second number.
A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
- receiving a request to perform a shuffle operation on a data stream that comprises a plurality of records that each include a key, the shuffle operation being an operation that groups the plurality of records in the data stream by key;
- producing an initial shuffler configuration, comprising assigning an initial key range that corresponds to the data stream to a first number of shufflers;
- determining, based on performance of the first number of shufflers, to reconfigure the initial shuffler configuration;
- and
- in response to determining to reconfigure the initial shuffler configuration, reconfiguring the initial shuffler configuration using a second number of shufflers, wherein the second number is different than the first number.
Dependencies are inferred from the claim text.