Mapping epigenetic surprisal data througth hadoop type distributed file systems

A computer program product for reducing an amount of epigenetic data representing epigenetic modifications of a genetic sequence of an organism using a file distributed system comprising a series of clusters coupled together, each cluster having at least one master node and a plurality of worker nodes, the computer program product comprising:

• one or more computer-readable, tangible storage devices;
• program instructions, stored on at least one of the one or more storage devices, to break a reference epigenetic map and epigenetic data from at least one point in time into blocks of data of a fixed size;
• program instructions, stored on at least one of the one or more storage devices, to distribute the blocks of data to the plurality of worker nodes within the clusters and replicating the blocks of data within each of the worker nodes;
• program instructions, stored on at least one of the one or more storage devices, to task the plurality of worker nodes to perform a map job comprising mapping the epigenetic data from at least a point in time by:
• comparing a subset of the epigenetic data representing epigenetic modifications of a genetic sequence of an organism to the mapped part of a genetic sequence of the reference epigenetic map, to find differences where epigenetic modifications of the genetic sequence of the organism are different from the mapped part of the genetic sequence of the reference epigenetic map;
• storing intermediate surprisal data in a key and value format in a repository of the cluster, the intermediate surprisal data comprising at least a starting location of the epigenetic modifications within the reference epigenetic map, and the modifications from the genetic sequence of the organism which are different from the reference epigenetic map, discarding modifications of the reference epigenetic map that are the same in the genetic sequence of the organism;
• and
• reporting the status of the task to map the reference epigenetic map to the epigenetic map at a specific point in time to the at least one master node of the cluster;
• when a worker node has reported a completion of the map job, program instructions, stored on at least one of the one or more storage devices, to task the worker node with a reduce job based on a specific key, comprising:
• the worker node shuffling the intermediate surprisal data between the worker node and a plurality of worker nodes of other clusters, based on the specific key;
• the worker node reducing the intermediate surprisal data to an output of surprisal data and associated metadata.

A system for reducing an amount of epigenetic data representing epigenetic modifications of a genetic sequence of an organism using a file distributed system comprising a series of clusters coupled together, each cluster having at least one master node and a plurality of worker nodes, the system comprising:

• one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices;
• program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to break a reference epigenetic map and epigenetic data from at least one point in time into blocks of data of a fixed size;
• program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to distribute the blocks of data to the plurality of worker nodes within the clusters and replicating the blocks of data within each of the worker nodes;
• program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to task the plurality of worker nodes to perform a map job comprising mapping the epigenetic data from at least a point in time by:
• comparing a subset of the epigenetic data representing epigenetic modifications of a genetic sequence of an organism to the mapped part of a genetic sequence of the reference epigenetic map, to find differences where epigenetic modifications of the genetic sequence of the organism are different from the mapped part of the genetic sequence of the reference epigenetic map;
• storing intermediate surprisal data in a key and value format in a repository of the cluster, the intermediate surprisal data comprising at least a starting location of the epigenetic modifications within the reference epigenetic map, and the modifications from the genetic sequence of the organism which are different from the reference epigenetic map, discarding modifications of the reference epigenetic map that are the same in the genetic sequence of the organism;
• and
• reporting the status of the task to map the reference epigenetic map to the epigenetic map at a specific point in time to the at least one master node of the cluster;
• when a worker node has reported a completion of the map job, program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to task the worker node with a reduce job based on a specific key, comprising:
• the worker node shuffling the intermediate surprisal data between the worker node and a plurality of worker nodes of other clusters, based on the specific key;
• the worker node reducing the intermediate surprisal data to an output of surprisal data and associated metadata.