Management of data segments for analytics queries
AssigneeLinkedIn Corporation
InventorsSanjay Dubey, Dhaval Patel, Praveen N. Naga, Volodymyr Zhabiuk
Life of this patent
Estimating the term…
Abstract
The disclosed embodiments provide a method and system for processing data. During operation, the system obtains a set of records, wherein each of the records comprises one or more metrics and at least one dimension associated with the one or more metrics. Next, the system creates a data segment comprising at least one of a forward index and an inverted index for a column in the records. The system then stores the data segment in network-accessible storage and assigns the data segment to a partition. Finally, the system enables querying of the data segment through a query node associated with the partition.
Claims20
A computer-implemented method for processing data, comprising:
- obtaining a set of records, wherein each of the records comprises one or more metrics and at least one dimension associated with the one or more metrics;
- sorting the records by values in a primary sort column of the records;
- creating a data segment comprising at least one of a forward index and an inverted index for a column in the sorted records;
- storing the data segment in network-accessible storage;
- assigning the data segment to a partition;
- enabling querying of the data segment through a query node associated with the partition;
- and
- upon identifying the data segment as a replacement for a previous version of the data segment in the partition,
- obtaining segment information for the data segment;
- based on the segment information notifying the query node of the replacement, thereby enabling the query node to replace the previous version of the data segment with the data segment.
A system for processing data, comprising:
- a data queue configured to provide a set of records, wherein each of the records comprises one or more metrics and at least one dimension associated with the one or more metrics;
- a segment-creation apparatus configured to:
- sort the records by values in a primary sort column of the records;
- and
- create a data segment comprising at least one of a forward index and an inverted index for a column in the sorted records;
- a segment-management apparatus configured to:
- store the data segment in network-accessible storage;
- and
- assign the data segment to a partition;
- and
- wherein upon identifying the data segment as a replacement for a previous version of the data segment in the partition, the segment-management apparatus is further configured to,
- obtain segment information for the data segment from the messaging service, and
- based on the segment information, provide a notification of the replacement to a query node associated with the partition,
- wherein the notification of the replacement enables the query node to replace the previous version of the data segment with the data segment.
A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for processing data, the method comprising:
- obtaining a set of records, wherein each of the records comprises one or more metrics and at least one dimension associated with the one or more metrics;
- sorting the records by values in a primary sort column of the records;
- creating a data segment comprising at least one of a forward index and an inverted index for a column in the sorted records;
- storing the data segment in network-accessible storage;
- assigning the data segment to a partition;
- enabling querying of the data segment through a query node associated with the partition;
- and
- upon identifying the data segment as a replacement for a previous version of the data segment in the partition,
- obtaining segment information for the data segment;
- based on the segment information notifying the query node of the replacement, thereby enabling the query node to replace the previous version of the data segment with the data segment.
Dependencies are inferred from the claim text.