Back to results
US 10,650,020 B1Granted
Analyzing transformations for preprocessing datasets
AssigneeTrifacta Inc.
InventorsVihang Jitendra Mehta, Seshadri Sadasivan Mahalingam, Philip John Vander Broek
Life of this patent
Estimating the term…
Abstract
A system analyzes transformations for processing datasets. The transformations may be used for build a transformation script for preprocessing data for analysis by big data analysis systems. The system receives a new transformation for analysis. The system determines a measure of an impact of the new transformation operation on a dataset. The system determines statistical information describing rows of the transformed dataset that are impacted by the new transformation. The system receives request to add the new transformation to the transformation script responsive to presenting the statistical information.
Claims24
A method for analyzing a dataset, the method comprising:
- receiving information identifying a dataset, the dataset comprising a plurality of rows, each row comprising values of one or more attributes;
- storing a transformation script for processing the dataset, the transformation script including a set of transformation operations;
- generating a sample of the dataset comprising a subset of the plurality of rows;
- applying the set of transformation operations of the transformation script to a sample of the dataset to obtain a transformed sample;
- receiving a new transformation operation, wherein the new transformation operation specifies a condition based on attribute values of a row of the dataset, and is configured to process the row of the dataset to generate an output value in response to the condition evaluating as true for the row;
- prior to adding the new transformation operation to the transformation script, presenting a user interface (UI) configured to present a preview of the impact of the transformation, the UI indicating a determined measure of impact of the new transformation operation on the transformed sample, wherein determining the measure of impact of the new transformation operation comprises:
- identifying rows of the transformed sample having attribute values that cause the condition specified by the new transformation operation to evaluate to true as rows impacted by the new transformation operation;
- determining statistical information describing rows of the transformed sample that are impacted by the new transformation operation;
- and
- presenting the statistical information describing rows of the transformed sample that are impacted by the new transformation operation;
- responsive to the statistical information indicating that the number of rows of the transformed sample impacted by the new transformation operation does not exceed a threshold number, updating the sample such that the transformed sample after applying the transformation script has at least the threshold number of rows impacted by the new transformation operation;
- responsive to presenting the statistical information describing rows of the transformed sample that are impacted by the new transformation, receiving a request to include the new transformation in the transformation script;
- and
- adding the new transformation operation to the transformation script responsive to the request to generate an updated transformation script.
11 more elements
A non-transitory computer readable storage medium storing instructions for:
- receiving information identifying a dataset, the dataset comprising a plurality of rows, each row comprising values of one or more attributes;
- storing a transformation script for processing the dataset, the transformation script including a set of transformation operations;
- generating a sample of the dataset comprising a subset of the plurality of rows;
- applying the set of transformation operations of the transformation script to a sample of the dataset to obtain a transformed sample;
- receiving a new transformation operation, wherein the new transformation operation specifies a condition based on attribute values of a row of the dataset, and is configured to process the row of the dataset to generate an output value in response to the condition evaluating as true for the row;
- prior to adding the new transformation operation to the transformation script, presenting a user interface (UI) configured to present a preview of the impact of the transformation, the UI indicating a determined measure of impact of the new transformation operation on the transformed sample, wherein determining the measure of impact of the new transformation operation comprises:
- identifying rows of the transformed sample having attribute values that cause the condition specified by the new transformation operation to evaluate to true as rows impacted by the new transformation operation;
- determining statistical information describing rows of the transformed sample that are impacted by the new transformation operation;
- and
- presenting the statistical information describing rows of the transformed sample that are impacted by the new transformation operation;
- responsive to the statistical information indicating that the number of rows of the transformed sample impacted by the new transformation operation does not exceed a threshold number, updating the sample such that the transformed sample after applying the transformation script has at least the threshold number of rows impacted by the new transformation operation;
- responsive to presenting the statistical information describing rows of the transformed sample that are impacted by the new transformation, receiving a request to include the new transformation in the transformation script;
- and
- adding the new transformation operation to the transformation script responsive to the request to generate an updated transformation script.
11 more elements
A computer-system comprising:
- a computer processor;
- and
- a computer readable non-transitory storage medium storing instructions that cause the computer processor to perform steps for:
- receiving information identifying a dataset, the dataset comprising a plurality of rows, each row comprising values of one or more attributes;
- generating a sample of the dataset comprising a subset of the plurality of rows;
- storing a transformation script including a set of transformation operations;
- applying the set of transformation operations of the transformation script to the sample of dataset to obtain a transformed sample;
- receiving a new transformation operation, wherein the new transformation operation specifies a condition based on attribute values of a row of the dataset, and is configured to process the row of the dataset to generate an output value in response to the condition evaluating as true for the row;
- prior to adding the new transformation operation to the transformation script, presenting a user interface (UI) configured to present a preview of the impact of the transformation, the UI indicating a determined measure of impact of the new transformation operation on the transformed sample, wherein determining the measure of impact of the new transformation operation comprises:
- identifying rows of the transformed sample having attribute values that cause the condition specified by the new transformation operation to evaluate to true as rows impacted by the new transformation operation;
- determining statistical information describing rows of the transformed sample that are impacted by the new transformation operation;
- and
- presenting the statistical information describing rows of the transformed sample that are impacted by the new transformation operation;
- responsive to the statistical information indicating that the number of rows of the transformed sample impacted by the new transformation operation does not exceed a threshold number, updating the sample such that the transformed sample after applying the transformation script has at least the threshold number of rows impacted by the new transformation operation;
- receiving a request for presenting rows impacted by the transformed sample;
- and
- selecting one or more rows impacted by the transformed sample and sending for presentation.
14 more elements
Dependencies are inferred from the claim text.