# US 2017/0286485 A1 · High Performance Query Processing and Data Analytics

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286485 A1
- Document ID: US20170286485A1
- Status: Published application
- Kind: A1
- Application number: 15/085,468
- Filed: 2016-03-30
- Published: 2017-10-05
- Earliest claimed: 2016-03-30
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Badrish Chandramouli; Raul Castro Fernandez; Abdul H. Quamar; Ahmed Eldawy; Jonathan D. Goldstein
- Web page: https://patentgrep.com/patent/US20170286485A1

## CPC

- G06F 16/24544: {Join order optimisation}
- G06F 16/245: Query processing
- G06F 16/24549: {Run-time optimisation}
- G06F 16/2477: {Temporal data queries}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

High performance query processing and data analytics can be performed across architecturally diverse scales, such as single core, multi-core and/or multi-nodes. The high performance query processing and data analytics can include a separation of query computation, keying data, and data movement and parallel computation, thereby enhancing the capabilities of the query processing and data analytics, while allowing the specification of complex forms of data parallel computation that may execute across real-time and offline. The decoupling of data movement and parallel computation, as described herein can improve query processing and data analytics speed, can provide for the optimization of searches in a plurality of computing environments, and can provide the ability to search through a larger space of execution plans.

## Claims

### Claim 1

A system comprising:
one or more processors; and
one or more computer-readable media having stored thereon computer-executable instructions, the computer-executable instructions responsive to execution by the one or more processors to configure a device to perform operations comprising:
receiving an identification of an input dataset;
receiving a transformation operation to be applied to the input dataset, wherein the transformation operation comprises one or more of:
a query comprising transferable computation logic;
a re-keying operation configured to associate a tuple in a shard of the one or more shards in the input dataset with an identifier; or
a data movement operation;
accessing the input dataset based at least in part on the identification, the input dataset comprising one or more shards of data;
applying the transformation operation to the input dataset; and
generating a resultant dataset based at least in part on a transformed dataset.

### Claim 2 (depends on 1)

The system as claim 1 recites, wherein the input dataset comprises data stored in a time-stamped order.

### Claim 3 (depends on 2)

The system as claim 2 recites, wherein the data movement operation comprises one or more of:
a broadcast join;
a multicast join; or
a re-distribution of contents of the one or more shards of data in the input dataset based at least in part on the identifier.

### Claim 4 (depends on 1)

The system as claim 1 recites, wherein the one or more shards of data are located over distributed memory.

### Claim 5 (depends on 1)

The system as claim 1 recites, wherein the input dataset comprises a first portion stored on a first machine of a distributed service platform and a second portion stored on a second machine of the distributed service platform, and wherein the transformation operation is performed by a virtual worker on the distributed service platform.

### Claim 6 (depends on 1)

The system as claim 1 recites, wherein the input dataset comprises a first portion located on a first core of a machine and a second portion located on a second core of the machine, and the transformation operation is based at least in part on a bandwidth between the first core of the machine and the second core of the machine.

### Claim 7 (depends on 1)

The system as claim 1 recites, the operations further comprising:
receiving an identification of a second input dataset;
accessing the second input dataset, the second input dataset comprising one or more shards of data; and
applying the transformation operation to the second input dataset,
wherein the transformation operation is a two-input transformation operation applied across a pair of shards in the input dataset and the second input dataset and wherein the resultant dataset comprises data from the input dataset and the second input dataset.

### Claim 8 (depends on 1)

The system as claim 1 recites, the operations further comprising:
receiving a query;
applying a query optimizer to the query;
generating a distributed physical plan; and
receiving a modification to the distributed physical plan.

### Claim 9

A method comprising:
defining a cluster of two or more machines, each of the two or more machines comprising at least one dataset;
requesting to establish a communication path with a virtual worker associated with the cluster;
associating an identifier with the virtual worker;
sending operations to the virtual worker via the communication path based at least in part on the identifier, the transformation operations including:
accessing a first dataset, the first dataset comprising at least two shards of data;
accessing a second dataset, the second dataset comprising at least two shards of data; and
generating a third dataset and a fourth dataset by redistributing data between one or more of:
the at least two shards of data on the first dataset;
the at least two shards of data on the second dataset; or
the first dataset and the second dataset.

### Claim 10 (depends on 9)

The method as claim 9 recites, wherein the sending the operations further comprises one or more of:
sending a query comprising transferable computation logic;
sending a re-keying operation configured to associate a tuple in a shard of the one or more shards in the input dataset with an identifier; or
sending a data movement operation.

### Claim 11 (depends on 9)

The method as claim 9 recites, wherein the sending the operations further comprises:
accessing a reference dataset, the reference dataset comprising at least two shards of data,
wherein the generating the third dataset and the fourth dataset further comprises joining the reference dataset to the first dataset and the second dataset.

### Claim 12 (depends on 11)

The method as claim 11 recites, wherein the joining the reference dataset to the first dataset and the second dataset comprises:
identifying a reference shard of the at least two shards of data associated with the reference dataset;
identifying a first shard of the at least two shards of data associated with the first dataset;
identifying a second shard of the at least two shards of data associated with the second dataset; and
sending data in the first shard to the first shard and the second shard.

### Claim 13 (depends on 11)

The method as claim 11 recites, wherein the joining the reference dataset to the first dataset and the second dataset comprises sending data in the at least two shards of data associated with the reference dataset to the at least two shards of data associated with the first dataset and the at least two shards of data associated with the second dataset.

### Claim 14 (depends on 9)

The method as claim 9 recites, wherein the at least two shards of data associated with the first dataset and the at least two shards of data associated with the second dataset comprise data stored in a time-stamped order.

### Claim 15 (depends on 9)

The method as claim 9 recites, further comprising:
storing, by the virtual worker, the third dataset and the fourth dataset in a disk dataset or an in-memory dataset; and
storing, by the virtual worker, metadata associated with the third dataset and the fourth dataset in a metadata table.

### Claim 16

A device comprising:
one or more processors;
one or more computer-readable media having thereon computer-executable instructions, the computer executable instructions responsive to execution to configure the device to perform operations comprising:
receiving an identification of an input dataset;
receiving a transformation operation to be applied to the input dataset, wherein the transformation operation comprises one or more of:
a query comprising transferable computation logic;
a re-keying operation configured to associate a tuple in a shard of the one or more shards in the input dataset with an identifier; or
a data movement operation;
accessing the input dataset based at least in part on the identification, the input dataset comprising one or more shards of data;
applying the transformation operation to the input dataset; and
generating a resultant dataset based at least in part on the transformed dataset.

### Claim 17 (depends on 16)

The device as claim 16 recites, wherein the input dataset comprises data stored in a time-stamped order.

### Claim 18 (depends on 16)

The device as claim 16 recites, wherein the data movement operation comprises one or more of:
a broadcast join;
a multicast join; or
a re-distribution of contents of the one or more shards of data in the input dataset based at least in part on the identifier.

### Claim 19 (depends on 16)

The device as claim 16 recites, wherein the input dataset comprises a first portion located on a first machine and a second portion located on a second machine, and the transformation operation comprises a redistribution of data between the first machine and the second machine.

### Claim 20 (depends on 16)

The device as claim 16 recites, the operations further comprising:
receiving a query;
applying a query optimizer to the query;
generating a distributed physical plan; and
receiving a modification to the distributed physical plan.
