US 2019/0102435 A1Published application

Heartbeat propagation in a distributed stream processing system

AssigneeOracle International Corporation

InventorsSandeep Bishnoi, Hoyong Park, Subhrajyoti Choudhury, Yathish Gatty, Devendra Jain, Santosh Kumar, Prabhu Thukkaram

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

An event processing system for distributing a continuous query, and processing events in an event stream, is disclosed. The system can synchronize timestamps across all partitions by adding a heartbeat propagator component propagating a heartbeat signal for silent partitions. The system can further include a timestamp tracker component that tracks the maximum observed timestamp across all partitions for each batch, and sends this to each partition. The system can distribute partitions of the events to the worker nodes, and instruct a heartbeat propagator component to generate a heartbeat for a first worker node corresponding to an event in a different partition. The system receives outputs from the worker nodes and combines these into a single result corresponding to a continuous query. The system can further implement a timestamp tracker to select a highest timestamp value of a previous batch, and send the value to the worker nodes.

Claims21

Dependent claims

A method for distributing a continuous query, comprising:

receiving, by a computing device, a plurality of events in a stream;
deploying, by the computing device, a query plan for a continuous query to a master node and a plurality of worker nodes, a respective worker node of the plurality of worker nodes implementing a respective heartbeat propagator component and a respective continuous query engine;
distributing a first partition of the plurality of events to the master node and to a first worker node in the plurality of worker nodes;
distributing a second partition of the plurality of events to the master node and to a second worker node in the plurality of worker nodes;
configuring the master node to instruct a first heartbeat propagator component implemented on the first worker node to propagate a first heartbeat for the first worker node, the first heartbeat corresponding to an event in the second partition;
receiving, from the first worker node, a first output identifying a first partition result, the first output being based at least in part on the first heartbeat and the query plan;
receiving, from the second worker node, a second output identifying a second partition result;
combining, by the computing device, the first output and the second output into a single result corresponding to the continuous query;
and
outputting the single result corresponding to the continuous query for a user.

7 more elements

A system, comprising:

a memory configured to store computer-executable instructions;
and
a processor configured to access the memory and execute the computer-executable instructions to perform a method for distributing a continuous query, the method comprising:
receiving a plurality of events in a stream;
deploying a query plan for a continuous query to a master node and a plurality of worker nodes, a respective worker node of the plurality of worker nodes implementing a respective heartbeat propagator component and a respective continuous query engine;
distributing a first partition of the plurality of events to the master node and to a first worker node in the plurality of worker nodes;
distributing a second partition of the plurality of events to the master node and to a second worker node in the plurality of worker nodes;
configuring the master node to instruct a first heartbeat propagator component implemented on the first worker node to propagate a first heartbeat for the first worker node, the first heartbeat corresponding to an event in the second partition;
receiving, from the first worker node, a first output identifying a first partition result, the first output being based at least in part on the first heartbeat and the query plan;
receiving, from the second worker node, a second output identifying a second partition result;
combining the first output and the second output into a single result corresponding to the continuous query;
and
outputting the single result corresponding to the continuous query for a user.

10 more elements

A non-transitory computer-readable medium storing computer-executable code that, when executed by a processor, cause the processor to perform a method for distributing a continuous query, the method comprising:

receiving a plurality of events in a stream;
deploying a query plan for a continuous query to a master node and a plurality of worker nodes, a respective worker node of the plurality of worker nodes implementing a respective heartbeat propagator component and a respective continuous query engine;
distributing a first partition of the plurality of events to the master node and to a first worker node in the plurality of worker nodes;
distributing a second partition of the plurality of events to the master node and to a second worker node in the plurality of worker nodes;
configuring the master node to instruct a first heartbeat propagator component implemented on the first worker node to propagate a first heartbeat for the first worker node, the first heartbeat corresponding to an event in the second partition;
receiving, from the first worker node, a first output identifying a first partition result, the first output being based at least in part on the first heartbeat and the query plan;
receiving, from the second worker node, a second output identifying a second partition result;
combining the first output and the second output into a single result corresponding to the continuous query;
and
outputting the single result corresponding to the continuous query for a user.

7 more elements

Dependencies are inferred from the claim text.