# US 2018/0025019 A1 · Platform for Analytic Applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0025019 A1 - Document ID: US20180025019A1 - Status: Published application - Kind: A1 - Application number: 15/218,296 - Filed: 2016-07-25 - Published: 2018-01-25 - Earliest claimed: 2016-07-25 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Shripad J. Nadgowda - Web page: https://patentgrep.com/patent/US20180025019A1 ## CPC - G06F 9/4843: {by program, e.g. task dispatcher, supervisor, operating system} - G06F 16/24561: {Intermediate data storage techniques for performance improvement} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and computer program products for analytic application platforms are provided herein. A computer-implemented method includes identifying one or more duplicate items of data in a dataset on which a computation task is to be performed by (i) segmenting the dataset into multiple segments and (ii) performing a deduplication operation on each of the multiple segments; removing the one or more duplicate items of data in the dataset from the computation task; performing the computation task on the remaining items of data in the dataset, wherein the remaining items of data comprise unique items of data in the dataset; aggregating (i) the results of the computation task and (ii) one or more memoized computation results corresponding to the one or more duplicate items of data to generate a complete computation result for the dataset; and outputting the complete computation result for the dataset to a user. ## Claims ### Claim 1 A computer-implemented method, comprising: identifying one or more duplicate items of data in a dataset on which a computation task is to be performed by (i) segmenting the dataset into multiple segments and (ii) performing a deduplication operation on each of the multiple segments; removing the one or more duplicate items of data in the dataset from the computation task; performing the computation task on the remaining items of data in the dataset, wherein the remaining items of data comprise unique items of data in the dataset; aggregating (i) the results of the computation task and (ii) one or more memoized computation results corresponding to the one or more duplicate items of data to generate a complete computation result for the dataset; and outputting the complete computation result for the dataset to a user; wherein the steps are carried out by at least one computing device. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the deduplication operation comprises performing a hash matching operation on each of the multiple segments of the dataset. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the deduplication operation comprises analyzing spatial and temporal locality across each of the multiple segments of the dataset. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein said performing a deduplication operation comprises reading a stored deduplication map. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, comprising: updating the stored deduplication map. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the multiple segments comprise multiple data blocks. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein each of the multiple data blocks comprises a data block of fixed size. ### Claim 8 (depends on 6) The computer-implemented method of claim 6, wherein each of the multiple data blocks comprises a data block of variable size. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, comprising: determining whether a computation task on each of the remaining items of data has been previously performed. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, comprising: aggregating the one or more memoized computation results corresponding to the one or more duplicate items of data. ### Claim 11 A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to: identify one or more duplicate items of data in a dataset on which a computation task is to be performed by (i) segmenting the dataset into multiple segments and (ii) performing a deduplication operation on each of the multiple segments; remove the one or more duplicate items of data in the dataset from the computation task; perform the computation task on the remaining items of data in the dataset, wherein the remaining items of data comprise unique items of data in the dataset; aggregate (i) the results of the computation task and (ii) one or more memoized computation results corresponding to the one or more duplicate items of data to generate a complete computation result for the dataset; and output the complete computation result for the dataset to a user. ### Claim 12 (depends on 11) The computer program product of claim 11, wherein the deduplication operation comprises performing a hash matching operation on each of the multiple segments of the dataset. ### Claim 13 (depends on 11) The computer program product of claim 11, wherein the deduplication operation comprises analyzing spatial and temporal locality across each of the multiple segments of the dataset. ### Claim 14 (depends on 11) The computer program product of claim 11, wherein said performing a deduplication operation comprises reading a stored deduplication map. ### Claim 15 (depends on 11) The computer program product of claim 11, wherein the multiple segments comprise multiple data blocks. ### Claim 16 (depends on 11) The computer program product of claim 11, wherein the program instructions executable by a computing device further cause the computing device to: determine whether a computation task on each of the remaining items of data has been previously performed. ### Claim 17 (depends on 11) The computer program product of claim 11, wherein the program instructions executable by a computing device further cause the computing device to: aggregate the one or more memoized computation results corresponding to the one or more duplicate items of data. ### Claim 18 A system comprising: a memory; and at least one processor coupled to the memory and configured for: identifying one or more duplicate items of data in a dataset on which a computation task is to be performed by (i) segmenting the dataset into multiple segments and (ii) performing a deduplication operation on each of the multiple segments; removing the one or more duplicate items of data in the dataset from the computation task; performing the computation task on the remaining items of data in the dataset, wherein the remaining items of data comprise unique items of data in the dataset; aggregating (i) the results of the computation task and (ii) one or more memoized computation results corresponding to the one or more duplicate items of data to generate a complete computation result for the dataset; and outputting the complete computation result for the dataset to a user. ### Claim 19 A computer-implemented method, comprising: identifying one or more duplicate data blocks in a dataset on which a computation task is to be performed by (i) segmenting the dataset into multiple segments and (ii) performing a deduplication operation on each of the multiple segments; removing the one or more duplicate data blocks in the dataset from the computation task; performing the computation task on the remaining items of data in the dataset, wherein the remaining items of data comprise unique items of data in the dataset; caching the results of the computation task for subsequent use as a memoized computation result; aggregating (i) the results of the computation task and (ii) one or more memoized computation results corresponding to the one or more duplicate data blocks to generate a complete computation result for the dataset; and outputting the complete computation result for the dataset to a display; wherein the steps are carried out by at least one computing device. ### Claim 20 (depends on 19) The computer-implemented method of claim 19, comprising: controlling the size of the memoized computation results.