Systems and methods for accelerating exploratory statistical analysis
AssigneePresident and Fellows of Harvard College
InventorsStratos IDREOS, Wasay ABDUL, Niv DAYAN
Life of this patent
Estimating the term…
Abstract
Embodiments of the invention utilize a “data canopy” that breaks statistical measures down to basic primitives for various data portions and stores the basic aggregates in a library within an in-memory data structure. When a queried statistical measure involves a basic aggregate stored in the library over a data portion that at least partially overlaps the data portion associated with the basic aggregate, the basic aggregate may be reused in the statistical computation of the queried measure.
Claims24
An apparatus for determining statistical properties of data, the apparatus comprising:
- a computer memory for storing (i) a user-supplied statistical query and (ii) a plurality of basic statistical primitives, each basic statistical primitive corresponding to a plurality of data chunks, each of the chunks corresponding to a smallest logical partition of data that includes consecutive values of data from a data structure;
- and
- a computer processor configured to:
- process the statistical query to identify at least one statistical computation and at least one data range corresponding to the at least one statistical computation;
- map the at least one data range to a subset of the data chunks;
- map the at least one statistical computation to a subset of the basic statistical primitives;
- perform the statistical computation;
- and
- respond to the statistical query with the statistical computation, wherein the statistical computation is based at least in part on the subset of data chunks and the subset of basic statistical primitives to thereby reduce a latency response time to the statistical query.
A method of determining statistical properties of data, the method comprising:
- storing, in a computer memory, a plurality of basic statistical primitives, each basic statistical primitive corresponding to a plurality of data chunks, each of the chunks corresponding to a smallest logical partition of data that includes consecutive values of data from a data structure;
- receiving a statistical query from a user;
- processing the received statistical query to thereby identify at least one statistical computation and at least one data range corresponding to the at least one statistical computation;
- computationally mapping the at least one data range to a subset of the data chunks;
- computationally mapping the at least one statistical computation to a subset of the basic statistical primitives;
- performing the statistical computation;
- and
- responding to the statistical query with the statistical computation, wherein the statistical computation is based at least in part on the subset of data chunks and the subset of basic statistical primitives to thereby reduce a latency response time to the statistical query.
Dependencies are inferred from the claim text.