US 9,805,101 B2Granted

Parallel data stream processing system

AssigneeeBay Inc.

InventorsGyanit Singh, Chi-Hsien Chiu, Neelakantan Sundaresan

Life of this patent

Estimating the term…

Abstract

A method and a system process a stream of data in parallel across a plurality of nodes. The log processing system has a log module, a query language module, and a query processing module. The log module receives and organizes the stream of data into a sequential and nested data structure. The query language operator module defines operators that operate on the sequential and nested data structure. The query processing module processes in parallel across a plurality of nodes a query based on an operator on the stream of data.

Claims16

Dependent claims

A log processing system, comprising:

a log module, implemented with at least one processor, configured to receive and organize a stream of data from a log of browsing activity of a user account into a sequential and nested data structure;
a query language operator module configured to define a relational and a sequential operator that both operate in parallel on the sequential and nested data structure;
and
a query processing module configured to receive a query related to the stream of data, to split a collection of tuples from the stream of data into a plurality of substreams of data without splitting the individual tuples, to process the query on the plurality of substreams across a plurality of nodes of the collection of tuples by operating the relational operator on the nested data structure of a tuple from the collection of tuples in parallel with operating the sequential operator on the sequential data structure of the tuple from the collection of tuples, wherein each respective individual tuple comprises an instance of one of:
a search activity by the user account, a bid activity by the user account and a view item activity by the user account,
the operators comprising an input operator, an output operator, a relational operator, and a pattern query,
the input operator configured to load session logs corresponding to a date range,
the output operator configured to save session logs in a storage device,
the relational operator configured to operate on the sequential and nested data structure based on relation conditions and extract sub-sequences,
the pattern query configured to operate on the sequential and nested data structure to identify sub-sequences that match an event pattern, the event pattern modeled as tuples and defined as a template that matches a set of contiguous events, the event pattern comprising respective instances of search activity that generated no results, the template describing order dependencies in the event pattern, data parameter conditions defining selection criteria for events, and context conditions identifying how events from a start of an event pattern and an end of an event pattern are broken into each pattern part.

7 more elements

A computer-implemented method, comprising:

receiving a stream of data from a log of browsing activity of a user account;
organizing the stream of data in a sequential and nested data structure;
defining a relational and a sequential operator that both operate in parallel on the sequential and nested data structure;
receiving a query related to the stream of data;
splitting a collection of tuples from the stream of data into a plurality of substream of data without splitting the individual tuples;
processing, using a hardware processor of a machine, the query on the plurality of substreams across a plurality of nodes of the collection of tuples by operating the relational operator on the nested data structure of a tuple from the collection of tuples in parallel with operating the sequential operator on the sequential data structure of the tuple from the collection of tuples, wherein each respective individual tuple comprises an instance of one of:
a search activity by the user account, a bid activity by the user account and a view item activity by the user account,
the operators comprising an input operator, an output operator, a relational operator, and a pattern query,
the input operator configured to load session logs corresponding to a date range,
the output operator configured to save session logs in a storage device,
the relational operator configured to operate on the sequential and nested data structure based on relation conditions and extract sub-sequences,
the pattern query configured to operate on the sequential and nested data structure to identify sub-sequences that match an event pattern, the event pattern modeled as tuples and defined as a template that matches a set of contiguous events, the event pattern comprising respective instances of search activity that generated no results, the template describing order dependencies in the event pattern, data parameter conditions defining selection criteria for events, and context conditions identifying how events from a start of an event pattern and an end of an event pattern are broken into each pattern part.

9 more elements

A non-transitory computer-readable storage medium storing a set of instructions that, when executed by a processor, cause the processor to perform operations, comprising:

receiving a stream of data from a log of browsing activity of a user account;
organizing the stream of data in a sequential and nested data structure;
defining a relational and a sequential operator that both operate in parallel on the sequential and nested data structure;
receiving a query related to the stream of data;
splitting a collection of tuples from the stream of data into a plurality of substream of data without splitting the individual tuples;
and
processing, using a hardware processor of a machine, the query on the plurality of substreams across a plurality of nodes of the collection of tuples by operating the relational operator on the nested data structure of a tuple from the collection of tuples in parallel with operating the sequential operator on the sequential data structure of the tuple from the collection of tuples, wherein each respective individual tuple comprises an instance of one of:
a search activity by the user account, a bid activity by the user account and a view item activity by the user account,
the operators comprising an input operator, an output operator, a relational operator, and a pattern query,
the input operator configured to load session logs corresponding to a date range,
the output operator configured to save session logs in a storage device,
the relational operator configured to operate on the sequential and nested data structure based on relation conditions and extract sub-sequences,
the pattern query configured to operate on the sequential and nested data structure to identify sub-sequences that match an event pattern, the event pattern modeled as tuples and defined as a template that matches a set of contiguous events, the event pattern comprising respective instances of search activity that generated no results, the template describing order dependencies in the event pattern, data parameter conditions defining selection criteria for events, and context conditions identifying how events from a start of an event pattern and an end of an event pattern are broken into each pattern part.

10 more elements

Dependencies are inferred from the claim text.

patentgrep

Loading patent…

Abstract

A log processing system, comprising:

a log module, implemented with at least one processor, configured to receive and organize a stream of data from a log of browsing activity of a user account into a sequential and nested data structure;
a query language operator module configured to define a relational and a sequential operator that both operate in parallel on the sequential and nested data structure;
and
a query processing module configured to receive a query related to the stream of data, to split a collection of tuples from the stream of data into a plurality of substreams of data without splitting the individual tuples, to process the query on the plurality of substreams across a plurality of nodes of the collection of tuples by operating the relational operator on the nested data structure of a tuple from the collection of tuples in parallel with operating the sequential operator on the sequential data structure of the tuple from the collection of tuples, wherein each respective individual tuple comprises an instance of one of:
a search activity by the user account, a bid activity by the user account and a view item activity by the user account,
the operators comprising an input operator, an output operator, a relational operator, and a pattern query,
the input operator configured to load session logs corresponding to a date range,
the output operator configured to save session logs in a storage device,
the relational operator configured to operate on the sequential and nested data structure based on relation conditions and extract sub-sequences,
the pattern query configured to operate on the sequential and nested data structure to identify sub-sequences that match an event pattern, the event pattern modeled as tuples and defined as a template that matches a set of contiguous events, the event pattern comprising respective instances of search activity that generated no results, the template describing order dependencies in the event pattern, data parameter conditions defining selection criteria for events, and context conditions identifying how events from a start of an event pattern and an end of an event pattern are broken into each pattern part.

7 more elements