Determining a degree of similarity of a subset of tabular data arrangements to subsets of graph data arrangements at ingestion into a data-driven collaborative dataset platform
Assigneedata.world, Inc.
InventorDavid Lee Griffith
Life of this patent
Estimating the term…
Abstract
Various techniques are described, including evaluating ingested data including a dataset to identify one or more links to other datasets stored in a graph, using a similarity determination algorithm to identify a degree of similarity between datasets to determine joinability of ingested datasets with graph-stored datasets, determining a ratio to determine whether to perform an overlap or coverage function, associating a subset of similarity matrices with a subset of graph data joined to the ingested dataset, and forming links in a column of data between the dataset and the another dataset of the ingested data based on the degree of similarity.
Claims20
A method, comprising:
- ingesting a dataset to form an ingested dataset;
- compressing the ingested dataset in accordance with at least one of one or more algorithmic hash functions to form a compressed data representation of the ingested data, the at least one of the one or more algorithmic hash functions including a plurality of differently-configured hash functions to generate a first subset of instances in a first state relative to a second subset of instances in a second state to specify a degree of similarity, the plurality of the differently-configured hash functions being configured to generate a plurality of hash values, at least one of the plurality of differently-configured hash functions being implemented based on a data type;
- identifying an indication associated with instructing computation of the degree of similarity between the ingested dataset and another dataset, the degree of similarity being used to join the dataset and the another dataset in a graph in response to executing instructions at one or more processors;
- determining a first ratio associated with an overlap function;
- determining a second ratio associated with a coverage function;
- identifying a metric configured to be used to determine the degree of similarity;
- associating a subset of similarity matrices with a subset of graph data joined to the ingested dataset;
- accessing the subset of similarity matrices, at least one subset of a similarity matrix is formed to identify a subset of relevant data associated with the another dataset disposed in a graph data arrangement, at least a portion of the another dataset in the graph data arrangement being formatted as one or more triple-based data formats, the similarity matrix and the degree of similarity being a function of the plurality of the hash values relating to a union of compressed ingested dataset data and compressed target data;
- forming a plurality of links among a column of data associated with the ingested dataset as the dataset and the another dataset of the ingested data, the plurality of links being determined based on the degree of similarity based on the compressed data representation;
- and
- receiving a query as a data operation based on ranked datasets identified based on the degree of similarity and other degrees of similarity,
- wherein a combined number of hash-derived attributes of the at least one of one or more algorithmic hash functions are configured to determine the degree of similarity based on multiple algorithmic hash functions.
A method, comprising:
- ingesting a dataset to form an ingested dataset;
- compressing the ingested dataset in accordance with at least one of one or more algorithmic hash functions to form a compressed data representation of the ingested data, the at least one of the one or more algorithmic hash functions including a plurality of differently-configured hash functions to generate a first subset of instances in a first state relative to a second subset of instances in a second state to specify a degree of similarity, the plurality of hash functions being configured to generate a plurality of hash values, at least one of the plurality of differently-configured hash functions being implemented based on a data type;
- evaluating the ingested data including a dataset and another dataset to identify one or more links to other datasets stored in a graph;
- using a similarity determination algorithm to identify the degree of similarity between the dataset, the another dataset, and the other datasets to determine joinability of the dataset, the another dataset, and the other datasets in the graph, the similarity determination algorithm including executable instructions processed at one or more processors;
- determining a ratio, the ratio being used to determine whether to perform a function;
- associating a subset of similarity matrices with a subset of graph data joined to the ingested dataset;
- accessing the subset of similarity matrices, at least one subset of a similarity matrix is formed to identify a subset of relevant data associated with the another dataset disposed in a graph data arrangement, at least a portion of the another dataset in the graph data arrangement being formatted as one or more triple-based data formats, the similarity matrix and the degree of similarity being a function of the plurality of the hash values relating to a union of compressed ingested dataset data and compressed target data;
- forming a plurality of links among a column of data associated with the ingested dataset as the dataset and the another dataset of the ingested data, the plurality of links being determined based on the degree of similarity based on the compressed data representation;
- and
- receiving a query as a data operation based on ranked datasets identified based on the degree of similarity and other degrees of similarity,
- wherein a combined number of hash-derived attributes of the at least one of one or more algorithmic hash functions are configured to determine the degree of similarity based on multiple algorithmic hash functions.
A system, comprising:
- a data repository including memory configured to store data associated with a graph stored in a data arrangement with which at least a portion is configured to store graph-based data arrangements;
- and
- a collaborative dataset consolidation system including one or more processors configured to ingest a dataset to form an ingested dataset, compress the ingested dataset in accordance with at least one of one or more algorithmic hash functions to form a compressed data representation of the ingested data, the at least one of the one or more algorithmic hash functions including a plurality of differently-configured hash functions to generate a first subset of instances in a first state relative to a second subset of instances in a second state to specify a degree of similarity, the plurality of hash functions being configured to generate a plurality of hash values, at least one of the plurality of differently-configured hash functions being implemented based on a data type, evaluate the ingested data including a dataset and another dataset to identify one or more links to other datasets stored in a graph, to use a similarity determination algorithm to identify the degree of similarity between the dataset, the another dataset, and the other datasets to determine joinability of the dataset, the another dataset, and the other datasets in the graph, to determine a ratio, the ratio being used to determine whether to perform a function, to associate a subset of similarity matrices with a subset of graph data joined to the ingested dataset, access the subset of similarity matrices, at least one subset of a similarity matrix is formed to identify a subset of relevant data associated with the another dataset disposed in a graph data arrangement, at least a portion of the another dataset in the graph data arrangement being formatted as one or more triple-based data formats, the similarity matrix and the degree of similarity being a function of the plurality of the hash values relating to a union of compressed ingested dataset data and compressed target data;
- to form a plurality of links among a column of data between the ingested dataset and the another dataset, the plurality of links being determined based on the degree of similarity based on the compressed data representation, and to receive a query as a data operation based on ranked datasets identified based on the degree of similarity and other degrees of similarity,
- wherein a combined number of hash-derived attributes of the at least one of one or more algorithmic hash functions are configured to determine the degree of similarity based on multiple algorithmic hash functions.
Dependencies are inferred from the claim text.