US 11,113,244 B1Granted

Integrated data pipeline

AssigneeA9.com, Inc.

InventorsGang Chen, Abraham Hossain Bagherjeiran, Chein-Hsin Liu

Life of this patent

Estimating the term…

Loading patent…

Abstract

An integrated data pipeline can take advantage of a streaming service, which can handle tasks such as automated redelivery, as well as a processing service, which can allocate workers on a task- or event-specific basis. Event data is aggregated and compressed for delivery by the streaming service. The streaming service can deliver the data asynchronously to the processing service, which can disaggregate and decompress the data to obtain the original data records. The type of event for each record can be determined to determine whether the data should be processed using online and/or offline processing. For online processing the appropriate fields are determined and data extracted to be passed to the online processing services. For offline processing the record data is concatenated sequentially into mini-batches, then compacted into larger batch files that are stored for subsequent offline processing.

Claims19

Dependent claims

A computer-implemented method, comprising:

receiving, from one or more data sources, event data for each of a plurality of events;
determining a maximum file size of the event data that is to be aggregated and compressed into each data block in a set of data blocks, wherein the maximum file size is determined as fixed or variable based at least in part on current workload or resource capacity;
aggregating, using at least two concurrently executing threads, the event data into the set of data blocks of up to a maximum file size;
compressing the data blocks using a block-based compression algorithm;
causing the data blocks to be transmitted asynchronously to a processing service, the data blocks being transmitted by a streaming service configured to provide multiple concurrent data streams for transmission of the data files;
allocating, by the processing service, one or more worker resources for performing tasks relating to the data blocks;
decompressing the data blocks to obtain individual data records for the plurality of events;
determining an event type for a specified record of individual data records, the event type being one of a bid, impression, or click;
determining at least one of online processing or offline processing to be performed for the specified record based at least in part upon the event type;
extracting, when the online processing is to be performed for the specified record, a specified set of data from the specified record and providing the specified set of data for the online processing;
and
providing, when the offline processing is to be performed for the specified record, a set of batched files of sequentially concatenated records from the individual data records to be stored for subsequent offline processing.

9 more elements

A computer-implemented method, comprising:

obtaining event data corresponding to occurrences of one or more types of events;
determining a maximum number of records of the event data that are to be aggregated and compressed into each data block in a set of data blocks, wherein the maximum number is determined as fixed or variable based at least in part on current workload or resource capacity;
generating, using at least two concurrently executing threads, the set of data blocks by aggregating and compressing the event data;
causing the data blocks to be transmitted asynchronously to a processing service;
determining, by the processing service, an event type for a specified record of a set of individual data records obtained upon decompressing the data blocks, the event type being one of a bid, impression, or click;
determining at least one of online processing or offline processing to be performed for the specified record based at least in part upon the event type;
extracting, when the online processing is to be performed for the specified record, specified data from the specified record and providing the specified set of data for the online processing;
and
storing, when the offline processing is to be performed for the specified record, a set of batched files of sequentially concatenated records from the individual data records for subsequent offline processing.

6 more elements

A system, comprising:

at least one processor;
and
memory including instructions that, when executed by the at least one processor, cause the system to:
obtain event data corresponding to occurrences of one or more types of events;
determine a maximum file size of the event data that is to be aggregated and compressed into each data block in a set of data blocks, wherein the maximum file size is determined as fixed or variable based at least in part on current workload or resource capacity;
generate, using at least two concurrently executing threads, the set of data blocks by aggregating and compressing the event data;
cause the data blocks to be transmitted asynchronously to a processing service;
determine, by the processing service, an event type for a specified record of a set of individual data records obtained upon decompressing the data blocks, the event type being one of a bid, impression, or click;
determine at least one of online processing or offline processing to be performed for the specified record based at least in part upon the event type;
extract, when the online processing is to be performed for the specified record, specified data from the specified record and providing the specified set of data for the online processing;
and
store, when the offline processing is to be performed for the specified record, a set of batched files of sequentially concatenated records from the individual data records for subsequent offline processing.

9 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/419,711Filed 30 Jan 2017
Granted: 7 Sep 2021
Term ends: Not estimatedThe source records don’t support an estimate.

Classification6 codes

G06F 16/1748
Redundancy elimination performed by the file system: de-duplication implemented within the file system, e.g. based on file segments
G06F 16/13
File systems; File servers: file access structures, e.g. distributed indices
G06F 16/1744
Redundancy elimination performed by the file system: using compression, e.g. sparse files
G06F 16/24568
Query execution: data stream processing; Continuous queries

Family

No related US applications are recorded in the indexed filing history.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key