# US 2021/0232592 A1 · Enhanced preparation and integration of data sets

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0232592 A1
- Document ID: US20210232592A1
- Status: Published application
- Kind: A1
- Application number: 17/156,112
- Filed: 2021-01-22
- Published: 2021-07-29
- Earliest claimed: 2020-01-23
- Assignees: MicroStrategy Incorporated
- Applicants: MicroStrategy Incorporated
- Inventors: Rixin Liao; Nannan Yu; Hongbo Liu; Scott Rigney; Benjamin Reyes
- Web page: https://patentgrep.com/patent/US20210232592A1

## CPC

- G06F 16/258: {Data format conversion from or to a database}
- G06F 16/2456: {Join operations}
- G06F 16/24568: {Data stream processing; Continuous queries}
- G06F 16/283: {Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP}
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 5/02: Knowledge representation; Symbolic representation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for enhanced preparation and integration of data sets. In some implementations, data indicating user input that identifies a first data set that includes streaming data and a second data set that includes non-streaming data is received. The first data set and the second data set are integrated to generate a hybrid data set. The data processing system provides access to the hybrid data set through a (i) non-streaming access channel that provides a periodically-refreshed summary of both the streaming data and the non-streaming data and (ii) a streaming access channel that provides a data stream based on combined data of the first data set and the second data set. One or more application programing interfaces are provided. The one or more application programming interfaces allow at least one client device to access the non-streaming access channel and the streaming access channel.

## Claims

### Claim 1

A method of integrating data streams and non-streaming data sets in a data processing system, the method comprising:
receiving, by one or more computers of the data processing system, data indicating user input that identifies (i) a first data set that includes streaming data and (ii) a second data set that includes non-streaming data;
integrating the first data set and the second data set to generate a hybrid data set, wherein the data processing system provides access to the hybrid data set through a (i) non-streaming access channel that provides a periodically-refreshed summary of both the streaming data and the non-streaming data and (ii) a streaming access channel that provides a data stream based on combined data of the first data set and the second data set; and
providing, by the one or more computers, one or more application programing interfaces, wherein the one or more application programming interfaces allow at least one client device to access the non-streaming access channel and the streaming access channel.

### Claim 2 (depends on 1)

The method of claim 1, wherein integrating the first data set and the second data set to generate the hybrid data set comprises generating a data cube that includes multiple processing pipelines for different levels of aggregation of component data sets, including data sets published in different portions of the data processing system,
wherein the component data sets include at least the first data set and the second data set,
wherein the non-streaming access channel is a first pipeline of the multiple processing pipelines, and
wherein the a streaming access channel is a second pipeline of the multiple processing pipelines.

### Claim 3 (depends on 1)

The method of claim 1, wherein integrating the first data set and the second data set to generate the hybrid data set comprises:
providing the first data set and the second data set as input to a machine learning model;
obtaining an output from the machine learning model, wherein the output of the machine learning model indicates a level of similarity between at least a portion of the first data set and at least a portion of the second data set; and
using the output of the machine learning model to integrate the first data set and the second data set into the hybrid data set, or to generate a recommendation for integrating the first data set and the second data set into the hybrid data set comprises using the level of similarity between the at least portion of the first data set and the at least portion of the second data set to (i) identify the at least portion of the first data set and the at least portion of the second data set, and (ii) determine that the at least portion of the first data set and the at least portion of the second data set should be integrated.

### Claim 4 (depends on 1)

The method of claim 1, wherein integrating the first data set and the second data set to generate the hybrid data set comprises:
providing the first data set and the second data set as input to a machine learning model;
obtaining an output from the machine learning model, wherein the output of the machine learning model indicates one or more recommended actions to integrate the first data set with the second data set into the hybrid data set; and
using the output of the machine learning model to integrate the first data set and the second data set into the hybrid data set, or to generate a recommendation for integrating the first data set and the second data set into the hybrid data set comprises:
performing the one or more recommended actions with the first data set and the second data set to generate the hybrid data set; or
generating a notification that includes the one or more recommended actions, and transmitting the notification to one or more client devices.

### Claim 5 (depends on 1)

The method of claim 1, wherein integrating the first data set and the second data set to generate the hybrid data set comprises:
accessing historical data that indicates how one or more other data sets have previously been integrated;
determining, from the historical data, one or more actions to integrate the first data set with the second data set; and
integrating at least a portion of the first data set with at least a portion of the second data set by performing the one or more actions, or generating a recommendation to integrate least a portion of the first data set with at least a portion of the second data set, wherein the recommendation includes an indication of the one or more actions.

### Claim 6 (depends on 5)

The method of claim 5, comprising generating signatures that interrelate previously performed actions in the historical data with particular characteristics of data sets that the previously performed actions were performed on, and storing the signatures in the historical data,
wherein determining, from the historical data, the one or more actions to integrate the first data set with the second data set comprises:
determining characteristics for the first data set and the second data set;
identifying one or more signatures in the historical data that correspond to the first data set and the second data set by comparing the characteristics for the first data set and the second data set to the characteristics corresponding to different signatures in the historical data; and
selecting the one or more actions in the historical data that correspond to the one or more signatures.

### Claim 7 (depends on 6)

The method of claim 6, wherein determining the characteristics for the first data set and the second data set comprises:
sampling the first data set to obtain a first subset of data;
sampling the second data set to obtain a second subset of data; and
analyzing the first subset of data and the second subset of data to determine the characteristics.

### Claim 8 (depends on 7)

The method of claim 7, wherein analyzing the first subset of data and the second subset of data comprises at least one of the following:
comparing the first subset of data and the second subset of data to determine a level of similarity between the first subset of data and the second subset of data;
determining a type of data in the first subset of data, the second subset of data, or both the first subset of data and the second subset of data;
determining a cardinality of the first subset of data, the second subset of data, or both the first subset of data and the second subset of data;
determining a distribute of the first subset of data, the second subset of data, or both the first subset of data and the second subset of data;
determining a range of values in the first subset of data, the second subset of data, or both the first subset of data and the second subset of data;
determining a variance of the first subset of data, the second subset of data, or both the first subset of data and the second subset of data;
determining a data size of the first subset of data, the second subset of data, or both the first subset of data and the second subset of data;
determining a data source of the first subset of data, the second subset of data, or both the first subset of data and the second subset of data;
determining a date and time associated with the first subset of data, the second subset of data, or both the first subset of data and the second subset of data; or
determining a user associated with the first subset of data, the second subset of data, or both the first subset of data and the second subset of data.

### Claim 9 (depends on 6)

The method of claim 6, wherein generating signatures that interrelate previously performed actions in the historical data with data sets having particular characteristics comprises generating signatures that specify one or more of the following:
a level of similarity between data sets;
a particular data type of a data set or a portion of a data set;
a cardinality of a data set or a portion of a data set;
a distribution of a data set or a portion of a data set;
a range of values in of a data set or a portion of a data set;
a variance of a data set or a portion of a data set;
a data size or data size range of a data set or a portion of a data set;
a data source of a data set or portion of a data set;
a date or time associated with a data set or a portion of a data set; or
a user associated with a data set or a portion of a data set.

### Claim 10 (depends on 5)

The method of claim 5, wherein determining the one or more actions to integrate the first data set with the second data set to generate the hybrid data set comprises determining a set of preprocessing actions to perform on at least one of the first data set or the second data set prior to integrating the first data set and the second data set, wherein the preprocessing actions may include performing one or more of the following:
removing a portion of data from the first data set;
removing a portion of data from the second data set;
modifying a format of at least a portion of data in the first data set;
modifying a format of at least a portion of data in the second data;
applying a series of transformation to the first data set;
applying a series of transformations to the second data set;
sampling the first data set; or
sampling the second data set.

### Claim 11 (depends on 5)

The method of claim 5, wherein determining the one or more actions to integrate the first data set with the second data set to generate the hybrid data set comprises determining to cache at least a portion of the first data set.

### Claim 12 (depends on 5)

The method of claim 5, wherein determining the one or more actions to integrate the first data set with the second data set to generate the hybrid data set comprises determining to join at least a portion of the first data set with at least a portion of the second data set.

### Claim 13 (depends on 5)

The method of claim 5, wherein determining the one or more actions to integrate the first data set with the second data set to generate the hybrid data set comprises determining to visualize at least a portion of the data sets, wherein at least a portion of the data sets includes one or more of the following:
at least a portion of the first data set;
at least a portion of the second data set;
a join of at least a portion of the first data set with at least a portion of the second data set; or
an overlay of at least a portion of the first data set and at least a portion of the second data set.

### Claim 14 (depends on 13)

The method of claim 13, wherein determining to visualize the at least portion of the data sets comprises determining to visualize the at least portion of the data sets as one or more histograms that describe different characteristics of the at least portion of the data sets.

### Claim 15 (depends on 5)

The method of claim 5, wherein determining the one or more actions to integrate the first data set with the second data set to generate the hybrid data set comprises determining to add (i) a structured data element to the first data set, the second data set, or to a data set resulting from a combination of the first data set and the second data set, (ii) one or more labels to the first data set, the second data set, or to a data set resulting from a combination of the first data set and the second data set, (iii) one or more metrics to the first data set, the second data set, or to a data set resulting from a combination of the first data set and the second data set, (iv) one or more sources to the first data set, the second data set, or to a data set resulting from a combination of the first data set and the second data set, (iv) one or more access control restrictions to the first data set, the second data set, or to a data set resulting from a combination of the first data set and the second data set, or (v) data from one or more data sets that are third-parties with respect to the first data set and the second data set to the first data set, the second data set, or to a data set resulting from a combination of the first data set and the second data set.

### Claim 16 (depends on 1)

The method of claim 1, wherein receiving the first data set that includes the streaming data comprises receiving a data set that includes one or more substantially-real-time streams.

### Claim 17 (depends on 1)

The method of claim 1, wherein receiving the first data set that includes the streaming data comprises receiving a data set that includes at least one of social media data, telemetry data, network traffic data, transaction data, or user input data.

### Claim 18 (depends on 1)

The method of claim 1, wherein receiving the second data set that includes the non-streaming data comprises receiving a data set that includes at least one of batch data or static data.

### Claim 19

A system comprising:
one or more computers;
one or more computer-readable media storing instructions that, when executed, cause the one or more computers to perform operations comprising:
receiving, by the one or more computers of a data processing system, data indicating user input that identifies (i) a first data set that includes streaming data and (ii) a second data set that includes non-streaming data;
integrating the first data set and the second data set to generate a hybrid data set, wherein the data processing system provides access to the hybrid data set through a (i) non-streaming access channel that provides a periodically-refreshed summary of both the streaming data and the non-streaming data and (ii) a streaming access channel that provides a data stream based on combined data of the first data set and the second data set; and
providing, by the one or more computers, one or more application programing interfaces, wherein the one or more application programming interfaces allow at least one client device to access the non-streaming access channel and the streaming access channel.

### Claim 20

One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving, by the one or more computers of a data processing system, data indicating user input that identifies (i) a first data set that includes streaming data and (ii) a second data set that includes non-streaming data;
integrating the first data set and the second data set to generate a hybrid data set, wherein the data processing system provides access to the hybrid data set through a (i) non-streaming access channel that provides a periodically-refreshed summary of both the streaming data and the non-streaming data and (ii) a streaming access channel that provides a data stream based on combined data of the first data set and the second data set; and
providing, by the one or more computers, one or more application programing interfaces, wherein the one or more application programming interfaces allow at least one client device to access the non-streaming access channel and the streaming access channel.
