Scaling high-level statistical languages to large, distributed datasets
AssigneeGoogle Inc.
InventorsMurray M. Stokely, Karl Millar
Life of this patent
Estimating the term…
Abstract
A system and method for performing large-scale data processing using a statistical programming language are disclosed. One or more high-level statistical operations may be received. The received high-level statistical operations may be dynamically translated into a graph of low-level data operations. The unnecessary operations may be removed and operations may be fused or chained together. Operations may then be grouped into distributed data processing operation. The low-level operations may then be run.
Claims18
A computer-implemented method for enabling efficient analysis of large-scale datasets comprising:
- receiving one or more first high-level statistical operations;
- dynamically translating the first high-level statistical operations into a graph of low-level data operations defining dependencies among the low-level data operations that will produce a result for the first high-level statistical operations when the low-level data operations are executed according to the graph;
- receiving, after dynamically translating the first high-level statistical operations, a second high-level statistical operation;
- adding an additional low-level data operation to the graph to define dependencies among the low-level data operations that will produce a result for the first high-level statistical operations and for the second high-level statistical operations;
- automatically transforming the graph of low-level data operations into an efficient graph by reducing the number of low-level data operations in the graph to define dependencies among the low-level data operations that will produce the same result as running the low-level data operations according to the graph;
- and
- running the low-level data operations according to the efficient graph to produce the result.
A system for enabling efficient analysis of large-scale datasets comprising:
- one or more processing devices;
- and
- one or more storage devices, the storage devices storing instructions that, when executed by the one or more processing devices, cause the processing devices to perform operations comprising:
- receiving one or more first high-level statistical operations;
- dynamically translating the first high-level statistical operations into a graph of low-level data operations defining dependencies among the low-level data operations that will produce a result for the first high-level statistical operations when the low-level data operations are executed according to the graph;
- receiving, after dynamically translating the first high-level statistical operations, a second high-level statistical operation;
- adding an additional low-level data operation to the graph to define dependencies among the low-level data operations that will produce a result for the first high-level statistical operations and for the second high-level statistical operations;
- automatically transforming the graph of low-level data operations into an efficient graph to define dependencies among the low-level data operations that will produce the same result as running the low-level data operations according to the graph;
- and
- running the low-level data operations according to the efficient graph to produce the result.
Dependencies are inferred from the claim text.