Platform for Analytic Applications

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:

• identify one or more duplicate items of data in a dataset on which a computation task is to be performed by (i) segmenting the dataset into multiple segments and (ii) performing a deduplication operation on each of the multiple segments;
• remove the one or more duplicate items of data in the dataset from the computation task;
• perform the computation task on the remaining items of data in the dataset, wherein the remaining items of data comprise unique items of data in the dataset;
• aggregate (i) the results of the computation task and (ii) one or more memoized computation results corresponding to the one or more duplicate items of data to generate a complete computation result for the dataset;
• and
• output the complete computation result for the dataset to a user.

A system comprising:

• a memory;
• and
• at least one processor coupled to the memory and configured for:
• identifying one or more duplicate items of data in a dataset on which a computation task is to be performed by (i) segmenting the dataset into multiple segments and (ii) performing a deduplication operation on each of the multiple segments;
• removing the one or more duplicate items of data in the dataset from the computation task;
• performing the computation task on the remaining items of data in the dataset, wherein the remaining items of data comprise unique items of data in the dataset;
• aggregating (i) the results of the computation task and (ii) one or more memoized computation results corresponding to the one or more duplicate items of data to generate a complete computation result for the dataset;
• and
• outputting the complete computation result for the dataset to a user.

A computer-implemented method, comprising:

• identifying one or more duplicate data blocks in a dataset on which a computation task is to be performed by (i) segmenting the dataset into multiple segments and (ii) performing a deduplication operation on each of the multiple segments;
• removing the one or more duplicate data blocks in the dataset from the computation task;
• performing the computation task on the remaining items of data in the dataset, wherein the remaining items of data comprise unique items of data in the dataset;
• caching the results of the computation task for subsequent use as a memoized computation result;
• aggregating (i) the results of the computation task and (ii) one or more memoized computation results corresponding to the one or more duplicate data blocks to generate a complete computation result for the dataset;
• and
• outputting the complete computation result for the dataset to a display;
• wherein the steps are carried out by at least one computing device.