US 12,613,840 B2Granted

Synthetic data generation

AssigneeSnowflake Inc.

InventorsGustavo Ajzenman, Khalid Zaman Bijon, Monica J. Holboke, Jeremy Marcell McGee

Life of this patent

Estimating the term…

Abstract

Various embodiments described herein provide for systems, methods, devices, instructions, and like for generating synthetic data. According to various embodiments, synthetic data generation comprises receiving input specifying one or more source tables and join key columns, and generating synthetic data that preserves statistical similarity and referential integrity among columns of the source data.

Claims20

Dependent claims

A system comprising:

at least one processor;
and
at least one memory storing instructions that cause the at least one processor to perform operations comprising:
creating, based on user input, a target table of a target database;
receiving a call to perform a synthetic data generation procedure with a set of input parameters, the set of input parameters comprising:
a first parameter that specifies a source table of a source database that stores original data;
and
a second parameter that specifies a set of join key columns of the source table;
performing the synthetic data generation procedure, the synthetic data generation procedure comprises:
accessing at least a portion of the original data from the source table;
generating synthetic data that comprises new data artificially generated based on the original data and the set of join key columns, the new data being artificially generated such that the new data is different from the original data while preserving statistical similarity to the original data and while preserving referential integrity of the new data among columns of the original data, the generating of the new data comprising:
mapping a set of unique key values, from the set of join key columns of the source table, to a set of unique integer values in a set of corresponding join key columns of an intermediate table:
transforming values in one or more non-join key columns of the source table to float values in one or more corresponding non-join key columns of the intermediate table;
determining a correlation matrix based on one or more statistical properties for one or more columns of the intermediate table;
generating a probabilistic model based on the one or more statistical properties and the correlation matrix;
and
sampling a random variable of the probabilistic model to generate the new data;
and
storing the synthetic data in the target table of the target database based on the set of join key columns;
using at least a portion of stored synthetic data from the target table to perform a training operation on an artificial intelligence model, the training operation yielding results similar to those obtained using the original data to perform the training operation on the artificial intelligence model;
and
computing one or more utility metrics that measure how well the artificial intelligence model trained on the synthetic data compares to another artificial intelligence model trained on the original data.

19 more elements

A method comprising:

creating, by at least one processor and based on user input, a target table of a target database;
receiving, by the at least one processor, a call to perform a synthetic data generation procedure with a set of input parameters, the set of input parameters comprising:
a first parameter that specifies a source table of a source database that stores original data;
and
a second parameter that specifies a set of join key columns of the source table;
performing, by the at least one processor, the synthetic data generation procedure, the synthetic data generation procedure comprises:
accessing at least a portion of the original data from the source table;
generating synthetic data that comprises new data artificially generated based on the original data and the set of join key columns, the new data being artificially generated such that the new data is different from the original data while preserving statistical similarity to the original data and while preserving referential integrity of the new data among columns of the original data, the generating of the new data comprising:
mapping a set of unique key values, from the set of join key columns of the source table, to a set of unique integer values in a set of corresponding join key columns of an intermediate table;
transforming values in one or more non-join key columns of the source table to float values in one or more corresponding non-join key columns of the intermediate table;
determining a correlation matrix based on one or more statistical properties for one or more columns of the intermediate table;
generating a probabilistic model based on the one or more statistical properties and the correlation matrix;
and
sampling a random variable of the probabilistic model to generate the new data;
and
storing the synthetic data in the target table of the target database based on the set of join key columns;
using, by the at least one processor, at least a portion of stored synthetic data from the target table to perform a training operation on an artificial intelligence model, the training operation yielding results similar to those obtained using the original data to perform the training operation on the artificial intelligence model;
and
computing, by the at least one processor, one or more utility metrics that measure how well the artificial intelligence model trained on the synthetic data compares to another artificial intelligence model trained on the original data.

16 more elements

A machine-storage medium, the machine-storage medium including instructions that when executed by a machine, cause the machine to perform operations comprising:

creating, based on user input, a target table of a target database;
receiving a call to perform a synthetic data generation procedure with a set of input parameters, the set of input parameters comprising:
a first parameter that specifies a source table of a source database that stores original data;
and
a second parameter that specifies a set of join key columns of the source table;
performing the synthetic data generation procedure, the synthetic data generation procedure comprises:
accessing at least a portion of the original data from the source table;
generating synthetic data that comprises new data artificially generated based on the original data and the set of join key columns, the new data being artificially generated such that the new data is different from the original data while preserving statistical similarity to the original data and while preserving referential integrity of the new data among columns of the original data, the generating of the new data comprising:
mapping a set of unique key values, from the set of join key columns of the source table, to a set of unique integer values in a set of corresponding join key columns of an intermediate table;
transforming values in one or more non-join key columns of the source table to float values in one or more corresponding non-join key columns of the intermediate table;
determining a correlation matrix based on one or more statistical properties for one or more columns of the intermediate table;
generating a probabilistic model based on the one or more statistical properties and the correlation matrix;
and
sampling a random variable of the probabilistic model to generate the new data;
and
storing the synthetic data in the target table of the target database based on the set of join key columns;
using at least a portion of stored synthetic data from the target table to perform a training operation on an artificial intelligence model, the training operation yielding results similar to those obtained using the original data to perform the training operation on the artificial intelligence model;
and
computing one or more utility metrics that measure how well the artificial intelligence model trained on the synthetic data compares to another artificial intelligence model trained on the original data.

16 more elements

Dependencies are inferred from the claim text.

patentgrep

Loading patent…

Abstract

A system comprising:

at least one processor;
and
at least one memory storing instructions that cause the at least one processor to perform operations comprising:
creating, based on user input, a target table of a target database;
receiving a call to perform a synthetic data generation procedure with a set of input parameters, the set of input parameters comprising:
a first parameter that specifies a source table of a source database that stores original data;
and
a second parameter that specifies a set of join key columns of the source table;
performing the synthetic data generation procedure, the synthetic data generation procedure comprises:
accessing at least a portion of the original data from the source table;
generating synthetic data that comprises new data artificially generated based on the original data and the set of join key columns, the new data being artificially generated such that the new data is different from the original data while preserving statistical similarity to the original data and while preserving referential integrity of the new data among columns of the original data, the generating of the new data comprising:
mapping a set of unique key values, from the set of join key columns of the source table, to a set of unique integer values in a set of corresponding join key columns of an intermediate table:
transforming values in one or more non-join key columns of the source table to float values in one or more corresponding non-join key columns of the intermediate table;
determining a correlation matrix based on one or more statistical properties for one or more columns of the intermediate table;
generating a probabilistic model based on the one or more statistical properties and the correlation matrix;
and
sampling a random variable of the probabilistic model to generate the new data;
and
storing the synthetic data in the target table of the target database based on the set of join key columns;
using at least a portion of stored synthetic data from the target table to perform a training operation on an artificial intelligence model, the training operation yielding results similar to those obtained using the original data to perform the training operation on the artificial intelligence model;
and
computing one or more utility metrics that measure how well the artificial intelligence model trained on the synthetic data compares to another artificial intelligence model trained on the original data.

19 more elements