US 10,896,178 B2Granted

High performance query processing and data analytics

AssigneeMicrosoft Technology Licensing, LLC

InventorsBadrish Chandramouli, Raul Castro Fernandez, Abdul H. Quamar, Ahmed Eldawy, Jonathan D. Goldstein

Life of this patent

Estimating the term…

Loading patent…

Abstract

High performance query processing and data analytics can be performed across architecturally diverse scales, such as single core, multi-core and/or multi-nodes. The high performance query processing and data analytics can include a separation of query computation, keying data, and data movement and parallel computation, thereby enhancing the capabilities of the query processing and data analytics, while allowing the specification of complex forms of data parallel computation that may execute across real-time and offline. The decoupling of data movement and parallel computation, as described herein can improve query processing and data analytics speed, can provide for the optimization of searches in a plurality of computing environments, and can provide the ability to search through a larger space of execution plans.

Claims20

Dependent claims

A system comprising:

one or more processors;
and
one or more non-transitory computer-readable media having stored thereon computer-executable instructions, the computer-executable instructions responsive to execution by the one or more processors to configure a device to perform operations comprising:
receiving an identification of an input dataset, wherein the input dataset comprises one or more shards of data;
re-keying the input dataset based on the identification;
redistributing, based on the re-keying, the input dataset into multiple datasets such that the one or more shards of data have a same identifier;
receiving a transformation operation to be applied to the input dataset, wherein the transformation operation comprises one or more of:
a query comprising transferable computation logic;
or
a data movement operation;
accessing the input dataset based at least in part on the identification;
applying the transformation operation to one of the multiple datasets;
and
generating a resultant dataset based at least in part on a transformed dataset.

11 more elements

A method comprising:

defining a cluster of two or more machines, each of the two or more machines comprising at least one dataset;
requesting to establish a communication path with a virtual worker associated with the cluster;
associating an identifier with the virtual worker;
sending operations to the virtual worker via the communication path based at least in part on the identifier, the transformation operations including:
accessing a first dataset, the first dataset comprising at least two shards of data;
accessing a second dataset, the second dataset comprising at least two shards of data;
re-keying the first dataset such that each of the at least two shards of data of the first dataset has an identifier based on a first topic;
re-keying the second dataset such that each of the at least two shards of data of the second dataset has an identifier based on a second topic;
and
generating a third dataset and a fourth dataset by redistributing data in the first dataset and the second set such that data shards in the third dataset have the same identifier and the data shards in the fourth data have the same identifier.

7 more elements

Dependencies are inferred from the claim text.