Structured cluster execution for data streams
AssigneeDatabricks, Inc.
InventorsMichael Paul Armbrust, Tathagata Das, Shi Xin, Matei Zaharia
Life of this patent
Estimating the term…
Abstract
A system for executing a streaming query includes an interface and a processor. The interface is configured to receive a logical query plan. The processor is configured to determine a physical query plan based at least in part on the logical query plan. The physical query plan comprises an ordered set of operators. Each operator of the ordered set of operators comprises an operator input mode and an operator output mode. The processor is further configured to execute the physical query plan using the operator input mode and the operator output mode for each operator of the query.
Claims20
A method for executing a streaming query, comprising:
- receiving, from a client device, a request to perform a streaming query on a data table, the streaming query specifying a set of operations on data of the data table;
- determining an output mode for the streaming query, the output mode specifying how results of the streaming query are written to an output table;
- and
- for each of one or more time increments, performing:
- receiving input data for the data table for a current time increment,
- executing the set of operations for the streaming query based on at least the input data for the current time increment to generate results for the set of operations,
- writing the results of to an output table for the current time increment based on the output mode for the streaming query, and
- providing the output table to the client device.
A non-transitory computer-readable storage medium comprising stored instructions executable by a processor, the instructions when executed causing the processor to:
- receive, from a client device, a request to perform a streaming query on a data table, the streaming query specifying a set of operations on data of the data table;
- determine an output mode for the streaming query, the output mode specifying how results of the streaming query is written to an output table;
- and
- for each of one or more time increments, execute instructions that cause the processor to:
- receive input data for the data table for a current time increment,
- execute the set of operations for the streaming query based on at least the input data for the current time increment to generate results for the set of operations,
- write the results of to an output table for the current time increment based on the output mode for the streaming query, and
- provide the output table to the client device.
A computer system, comprising:
- a computer processor;
- and
- a memory comprising stored instructions that when executed by the computer processor causes the computer system to:
- receive, from a client device, a request to perform a streaming query on a data table, the streaming query specifying a set of operations on data of the data table;
- determine an output mode for the streaming query, the output mode specifying how results of the streaming query is written to an output table;
- and
- for each of one or more time increments, repeatedly:
- receive input data for the data table for a current time increment,
- execute the set of operations for the streaming query based on at least the input data for the current time increment to generate results for the set of operations,
- write the results of to an output table for the current time increment based on the output mode for the streaming query, and
- provide the output table to the client device.
Dependencies are inferred from the claim text.