Back to results
US 12,717,810 B2Granted
System and method for finding segments in a dataset within a data analytics environment
AssigneeOracle International Corporation
InventorMikhail Shmulyan
Life of this patent
Estimating the term…
Abstract
In accordance with an embodiment, described herein is a system and method for determining or finding segments in a dataset within a data analytics environment. Upon selection of a segment of interest from a dataset, the systems and methods can determine, based on scoring, one or more other segments of data that are determined to be likely of interest. The determination and finding of segments of a dataset can allow a user to explore dataset filters, which can maximize or minimize an average of a numeric column, or the frequency of certain attribute of an ordinal column. The systems and methods can utilize, for example, a combination of brute force, smart sampling, or other optimization techniques for the determination of segments.
Claims20
A system for data analytics, including automatic determination of segments of interest associated with datasets, comprising:
- a computer comprising a microprocessor and a memory and a data analytics environment provided thereon and operable to display data visualizations associated with sets of data, the data analytics environment comprising a control plane and a data plane, wherein the data analytics environment is communicatively coupled to a data warehouse instance, the data warehouse instance being provisioned by the control plane in association with a user and including a schema within the data warehouse instance associated with the user, based upon instructions received at the control plane;
- wherein, upon selection of a dataset from the data warehouse instance for visualization, the selection being based upon an instruction received at the control plane of the data analytics environment, the dataset comprising a plurality of string attribute columns, a plurality of numerical attribute columns, a target numerical column, and a list of target column values, wherein the computer comprising the microprocessor and the memory, at the data plane:
- automatically determines, as a plurality of segments of the data, one or more groupings of data associated with the dataset,
- said automatically determining of the plurality of segments of the data utilizing sampling to downsample the dataset,
- said automatically determining of the plurality of segments of data further comprising converting each value of the plurality of string attribute columns and each value of the plurality of numerical attribute columns to an integer and evaluating, utilizing a flat hash map structure executed via a plurality of threads utilizing the microprocessor, combinations of converted string attribute columns and converted attribute values utilizing a determined plurality of predicate filters;
- wherein the computer further operates to determine filters which select a subset of the dataset where the frequency of one of the target column values of the target column differs significantly compared to a global frequency for the same value calculated for the dataset;
- wherein the computer operates to evaluate possible combinations of attribute columns and attribute values;
- wherein based on a total number of rows and columns, the computer operates to estimate and limit a computing cost of processing within a limit by limiting a subset of the attribute columns while looking for segments by scoring each column by its ability to generate segments for further splitting and how much of a driver it is for the target numerical column, and selecting best columns using this score;
- wherein the determined plurality of segments of data are pruned according to a first filter, wherein the pruning according to a first filter discards a first set of the determined plurality of segments of data, each of the first set of the determined plurality of segments of data comprising a number of rows less than a predetermined number of rows;
- wherein the determined plurality of segments of data are pruned according to a first filter, wherein the pruning according to a first filter discards a first set of the determined plurality of segments of data, each of the first set of the determined plurality of segments of data comprising a number of rows less than a predetermined number of rows;
- wherein the determined plurality of segments of data are further pruned according to a second filter, wherein the pruning according to the second filter discards a second set of the determined plurality of segments of data, each of the second set of the determined plurality of segments of data comprising a score less than a predetermined minimum score;
- wherein the pruned determined plurality of segments of data are displayed via a user interface of the data analytics environment;
- and
- wherein, based upon interactions at the user interface, the display of the pruned determined plurality of segments of data is updated according to a selection of an attribute of the target numerical column.
11 more elements
A method for data analytics including automatically determining segments of interest associated with datasets, comprising:
- providing, at a computer comprising a microprocessor and a memory, a data analytics environment operable to display data visualizations associated with datasets, the data analytics environment comprising a control plane and a data plane, wherein the data analytics environment is communicatively coupled to a data warehouse instance, the data warehouse instance being provisioned by the control plane in association with a user and including a schema within the data warehouse instance associated with the user, based upon instructions received at the control plane;
- upon selection of a dataset from the data warehouse instance for visualization, the selection being based upon an instruction received at the control plane of the data analytics environment, the dataset comprising a plurality of string attribute columns, a plurality of numerical attribute columns, a target numerical column, and a list of target column values,
- automatically determining, as a plurality of segments of the data, one or more groupings of data associated with the dataset,
- said automatically determining of the plurality of segments of the data utilizing sampling to downsample the dataset,
- said automatically determining of the plurality of segments of data further comprising converting each value of the plurality of string attribute columns and each value of the plurality of numerical attribute columns to an integer and evaluating, utilizing a flat hash map structure executed via a plurality of threads utilizing the microprocessor, combinations of converted string attribute columns and converted attribute values utilizing a determined plurality of predicate filters;
- determining filters which select a subset of the dataset where the frequency of one of the target column values of the target column differs significantly compared to a global frequency for the same value calculated for the dataset;
- evaluating possible combinations of attribute columns and attribute values;
- based on a total number of rows and columns, estimating and limiting a computing cost of processing within a limit by limiting a subset of the attribute columns while looking for segments by scoring each column by its ability to generate segments for further splitting and how much of a driver it is for the target numerical column, and selecting best columns using this score;
- pruning the determined plurality of segments of data according to a first filter, wherein the pruning according to a first filter discards a first set of the determined plurality of segments of data, each of the first set of the determined plurality of segments of data comprising a number of rows less than a predetermined number of rows;
- pruning the determined plurality of segments of data according to a second filter, wherein the pruning according to the second filter discards a second set of the determined plurality of segments of data, each of the second set of the determined plurality of segments of data comprising a score less than a predetermined minimum score;
- displaying, via a user interface of the data analytics environment, the pruned determined plurality of segments of data;
- and updating, based upon interactions at the user interface, the display of the pruned determined plurality of segments of data according to a selection of an attribute of the target numerical column.
9 more elements
A non-transitory computer readable storage medium, including instructions stored thereon which when read and executed by one or more computers cause the one or more computers to perform a method comprising:
- providing, at a computer comprising a microprocessor and a memory, a data analytics environment operable to display data visualizations associated with datasets, the data analytics environment comprising a control plane and a data plane, wherein the data analytics environment is communicatively coupled to a data warehouse instance, the data warehouse instance being provisioned by the control plane in association with a user and including a schema within the data warehouse instance associated with the user, based upon instructions received at the control plane;
- upon selection of a dataset from the data warehouse instance for visualization, the selection being based upon an instruction received at the control plane of the data analytics environment, the dataset comprising a plurality of string attribute columns, a plurality of numerical attribute columns, a target numerical column, and a list of target column values,
- automatically determining, as a plurality of segments of the data, one or more groupings of data associated with the dataset,
- said automatically determining of the plurality of segments of the data utilizing sampling to downsample the dataset,
- said automatically determining of the plurality of segments of data further comprising converting each value of the plurality of string attribute columns and each value of the plurality of numerical attribute columns to an integer and evaluating, utilizing a flat hash map structure executed via a plurality of threads utilizing the microprocessor, combinations of converted string attribute columns and converted attribute values utilizing a determined plurality of predicate filters;
- determining filters which select a subset of the dataset where the frequency of one of the target column values of the target column differs significantly compared to a global frequency for the same value calculated for the dataset;
- evaluating possible combinations of attribute columns and attribute values;
- based on a total number of rows and columns, estimating and limiting a computing cost of processing within a limit by limiting a subset of the attribute columns while looking for segments by scoring each column by its ability to generate segments for further splitting and how much of a driver it is for the target numerical column, and selecting best columns using this score;
- pruning the determined plurality of segments of data according to a first filter, wherein the pruning according to a first filter discards a first set of the determined plurality of segments of data, each of the first set of the determined plurality of segments of data comprising a number of rows less than a predetermined number of rows;
- pruning the determined plurality of segments of data according to a second filter, wherein the pruning according to the second filter discards a second set of the determined plurality of segments of data, each of the second set of the determined plurality of segments of data comprising a score less than a predetermined minimum score;
- displaying, via a user interface of the data analytics environment, the pruned determined plurality of segments of data;
- and updating, based upon interactions at the user interface, the display of the pruned determined plurality of segments of data according to a selection of an attribute of the target numerical column.
9 more elements
Dependencies are inferred from the claim text.