# US 2014/0156683 A1 · Integrating event processing with map-reduce

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0156683 A1
- Document ID: US20140156683A1
- Status: Published application
- Kind: A1
- Application number: 14/079,538
- Filed: 2013-11-13
- Published: 2014-06-05
- Earliest claimed: 2012-12-05
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Alexandre de Castro Alves
- Web page: https://patentgrep.com/patent/US20140156683A1

## CPC

- G06F 16/24568: {Data stream processing; Continuous queries}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Large quantities of data can be processed and/or queried relatively quickly using a combination of continuous event processing and a Map-Reduce algorithmic tool. The continuous event processor can continuously produce real-time results by merging (a) CQL query results from events received since a currently executing Map-Reduce job was started with (b) a most recent query result produced by a most recently completed Map-Reduce job. When the currently executing Map-Reduce job completes, its query result can be stored and made accessible to the continuous event processor, and a new Map-Reduce job can be started relative to event data that has grown in size since the execution of the last Map-Reduce job. The Map-Reduce algorithmic tool provides a convenient mechanism for analyzing and processing large quantities of data.

## Claims

### Claim 1

A computer-implemented method comprising:
starting a first batch operation that executes a first query relative to event data that has been stored in a persistent data store as of a first time;
while the first batch operation is executing, continuously executing a second query relative to event data that is continuously being received from a data stream;
while the first batch operation is executing, continuously merging results of the second query with a result of a second batch operation that has completed;
continuously outputting results of the merging; and
in response to a completion of the first batch operation, replacing the result of the second batch operation with a result of the first batch operation.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
while the first batch operation is executing, continuously storing, in the persistent data store, the event data that is continuously being received from the data stream;
in response to the completion of the first batch operation, starting a third batch operation that executes the first query relative to event data that has been stored in the persistent data store as of a second time that is later than the first time;
while the third batch operation is executing, continuously executing the second query relative to the event data that is continuously being received from a data stream;
while the third batch operation is executing, continuously merging results of the second query with a result of the first batch operation that has completed;
continuously outputting results of the merging of the second query with the result of the first batch operation that has completed; and
in response to a completion of the third batch operation, replacing the result of the first batch operation with a result of the third batch operation.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the first batch operation is a Map-Reduce operation, and wherein the second query performs same operations as the first query.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a continuous query language (CQL) query as the second query; and
automatically generating, based on the CQL query, a Map-Reduce program that performs operations of the CQL query.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a continuous query language (CQL) query as the second query;
automatically parsing the CQL query;
based on the parsing, automatically separating the CQL query into a first set of operators;
automatically generating, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and
automatically generating, based on the Pig Latin operators, a Map-Reduce program that performs the first batch operation.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a continuous query language (CQL) query as the second query;
automatically parsing the CQL query;
based on the parsing, automatically separating the CQL query into a first set of operators;
automatically generating, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and
automatically generating a directed graph of nodes including operators from the first set, operators from the second set, edges from nodes representing operators from the first set to nodes representing operators from the second set, and edges from nodes representing operators form the second set to operators from the first set.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving a continuous query language (CQL) query as the second query;
automatically parsing the CQL query;
based on the parsing, automatically separating the CQL query into a first set of operators;
automatically generating, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators;
automatically generating a directed graph of nodes including operators from the first set and operators from the second set; and
automatically merging two or more nodes in the directed graph based on similarities between operations performed by the two or more nodes.

### Claim 8

A computer-readable memory storing particular instructions to cause one or more processors to perform operators, the particular instructions comprising:
instructions to cause the one or more processors to start a first batch operation that executes a first query relative to event data that has been stored in a persistent data store as of a first time;
instructions to cause the one or more processors to continuously execute, while the first batch operation is executing, a second query relative to event data that is continuously being received from a data stream;
instructions to cause the one or more processors to continuously merge, while the first batch operation is executing, results of the second query with a result of a second batch operation that has completed;
instructions to cause the one or more processors to continuously output results of the merging; and
instructions to cause the one or more processors to replace, in response to a completion of the first batch operation, the result of the second batch operation with a result of the first batch operation.

### Claim 9 (depends on 8)

The computer-readable memory of claim 8, wherein the particular instructions further comprise:
instructions to cause the one or more processors to continuously store, while the first batch operation is executing, and in the persistent data store, the event data that is continuously being received from the data stream;
instructions to cause the one or more processors to start, in response to the completion of the first batch operation, a third batch operation that executes the first query relative to event data that has been stored in the persistent data store as of a second time that is later than the first time;
instructions to cause the one or more processors to continuously execute, while the third batch operation is executing, the second query relative to the event data that is continuously being received from a data stream;
instructions to cause the one or more processors to continuously merge, while the third batch operation is executing, results of the second query with a result of the first batch operation that has completed;
instructions to cause the one or more processors to continuously output results of the merging of the second query with the result of the first batch operation that has completed; and
instructions to cause the one or more processors to replace, in response to a completion of the third batch operation, the result of the first batch operation with a result of the third batch operation.

### Claim 10 (depends on 8)

The computer-readable memory of claim 8, wherein the first batch operation is a Map-Reduce operation, and wherein the second query performs same operations as the first query.

### Claim 11 (depends on 8)

The computer-readable memory of claim 8, wherein the particular instructions further comprise:
instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query; and
instructions to cause the one or more processors to automatically generate, based on the CQL query, a Map-Reduce program that performs operations of the CQL query.

### Claim 12 (depends on 8)

The computer-readable memory of claim 8, wherein the particular instructions further comprise:
instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query;
instructions to cause the one or more processors to automatically parse the CQL query;
instructions to cause the one or more processors to automatically separate, based on the parsing, the CQL query into a first set of operators;
instructions to cause the one or more processors to automatically generate, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and
instructions to cause the one or more processors to automatically generate, based on the Pig Latin operators, a Map-Reduce program that performs the first batch operation.

### Claim 13 (depends on 8)

The computer-readable memory of claim 8, wherein the particular instructions further comprise:
instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query;
instructions to cause the one or more processors to automatically parse the CQL query;
instructions to cause the one or more processors to automatically separate, based on the parsing, the CQL query into a first set of operators;
instructions to cause the one or more processors to automatically generate, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and
instructions to cause the one or more processors to automatically generate a directed graph of nodes including operators from the first set, operators from the second set, edges from nodes representing operators from the first set to nodes representing operators from the second set, and edges from nodes representing operators form the second set to operators from the first set.

### Claim 14 (depends on 8)

The computer-readable memory of claim 8, wherein the particular instructions further comprise:
instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query;
instructions to cause the one or more processors to automatically parse the CQL query;
instructions to cause the one or more processors to automatically separate, based on the parsing, the CQL query into a first set of operators;
instructions to cause the one or more processors to automatically generate, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators;
instructions to cause the one or more processors to automatically generate a directed graph of nodes including operators from the first set and operators from the second set; and
instructions to cause the one or more processors to automatically merge two or more nodes in the directed graph based on similarities between operations performed by the two or more nodes.

### Claim 15

A system comprising:
one or more processors; and
a computer-readable memory storing particular instructions to cause the one or more processors to perform operators, the particular instructions comprising:
instructions to cause the one or more processors to start a first batch operation that executes a first query relative to event data that has been stored in a persistent data store as of a first time;
instructions to cause the one or more processors to continuously execute, while the first batch operation is executing, a second query relative to event data that is continuously being received from a data stream;
instructions to cause the one or more processors to continuously merge, while the first batch operation is executing, results of the second query with a result of a second batch operation that has completed;
instructions to cause the one or more processors to continuously output results of the merging; and
instructions to cause the one or more processors to replace, in response to a completion of the first batch operation, the result of the second batch operation with a result of the first batch operation.

### Claim 16 (depends on 15)

The system of claim 15, wherein the particular instructions further comprise:
instructions to cause the one or more processors to continuously store, while the first batch operation is executing, and in the persistent data store, the event data that is continuously being received from the data stream;
instructions to cause the one or more processors to start, in response to the completion of the first batch operation, a third batch operation that executes the first query relative to event data that has been stored in the persistent data store as of a second time that is later than the first time;
instructions to cause the one or more processors to continuously execute, while the third batch operation is executing, the second query relative to the event data that is continuously being received from a data stream;
instructions to cause the one or more processors to continuously merge, while the third batch operation is executing, results of the second query with a result of the first batch operation that has completed;
instructions to cause the one or more processors to continuously output results of the merging of the second query with the result of the first batch operation that has completed; and
instructions to cause the one or more processors to replace, in response to a completion of the third batch operation, the result of the first batch operation with a result of the third batch operation.

### Claim 17 (depends on 15)

The system of claim 15, wherein the first batch operation is a Map-Reduce operation, and wherein the second query performs same operations as the first query.

### Claim 18 (depends on 15)

The system of claim 15, wherein the particular instructions further comprise:
instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query; and
instructions to cause the one or more processors to automatically generate, based on the CQL query, a Map-Reduce program that performs operations of the CQL query.

### Claim 19 (depends on 15)

The system of claim 15, wherein the particular instructions further comprise:
instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query;
instructions to cause the one or more processors to automatically parse the CQL query;
instructions to cause the one or more processors to automatically separate, based on the parsing, the CQL query into a first set of operators;
instructions to cause the one or more processors to automatically generate, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and
instructions to cause the one or more processors to automatically generate, based on the Pig Latin operators, a Map-Reduce program that performs the first batch operation.

### Claim 20 (depends on 15)

The system of claim 15, wherein the particular instructions further comprise:
instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query;
instructions to cause the one or more processors to automatically parse the CQL query;
instructions to cause the one or more processors to automatically separate, based on the parsing, the CQL query into a first set of operators;
instructions to cause the one or more processors to automatically generate, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and
instructions to cause the one or more processors to automatically generate a directed graph of nodes including operators from the first set, operators from the second set, edges from nodes representing operators from the first set to nodes representing operators from the second set, and edges from nodes representing operators form the second set to operators from the first set.