Mapping surprisal data througth hadoop type distributed file systems

A computer program product for reducing an amount of data representing a genetic sequence of an organism using a file distributed system comprising a series of clusters coupled together, each cluster having at least one master node and a plurality of worker nodes, the computer program product comprising:

• one or more computer-readable, tangible storage devices;
• program instructions, stored on at least one of the one or more storage devices, to break a surprisal data filter and an uncompressed genetic sequence into blocks of data of a fixed size;
• program instructions, stored on at least one of the one or more storage devices, to distribute the blocks of data to the plurality of worker nodes within the clusters and replicating the blocks of data within each of the worker nodes;
• program instructions, stored on at least one of the one or more storage devices, to task the plurality of worker nodes to perform a map job comprising mapping the surprisal data filter relative to the uncompressed genetic sequence by:
• comparing nucleotides of the genetic sequence of the organism to nucleotides of the assigned part of the surprisal data filter, to find differences where nucleotides of the genetic sequence of the organism are different from the nucleotides of the surprisal data filter;
• storing intermediate surprisal data in a key and value format in a repository of the cluster, the intermediate surprisal data comprising at least a starting location of the differences within the surprisal data filter, and the nucleotides from the genetic sequence of the organism which are different from the nucleotides the surprisal data filter, discarding sequences of nucleotides that are the same in the genetic sequence of the organism;
• and
• reporting the status of the task to map the surprisal data filter to the uncompressed genetic sequence to the at least one master node of the cluster;
• when a worker node has reported a completion of the map job, program instructions, stored on at least one of the one or more storage devices, to task the worker node with a reduce job based on a specific key, comprising:
• the worker node shuffling the intermediate surprisal data between the worker node and a plurality of worker nodes of other clusters, based on the specific key;
• the worker node reducing the intermediate surprisal data to an output of surprisal data and associated metadata.

A system for reducing an amount of data representing a genetic sequence of an organism using a file distributed system comprising a series of clusters coupled together, each cluster having at least one master node and a plurality of worker nodes, the system comprising:

• one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices;
• program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to break a surprisal data filter and an uncompressed genetic sequence into blocks of data of a fixed size;
• program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to distribute the blocks of data to the plurality of worker nodes within the clusters and replicating the blocks of data within each of the worker nodes;
• program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to task the plurality of worker nodes to perform a map job comprising mapping the surprisal data filter relative to the uncompressed genetic sequence by:
• comparing nucleotides of the genetic sequence of the organism to nucleotides of the assigned part of the surprisal data filter, to find differences where nucleotides of the genetic sequence of the organism are different from the nucleotides of the surprisal data filter;
• storing intermediate surprisal data in a key and value format in a repository of the cluster, the intermediate surprisal data comprising at least a starting location of the differences within the surprisal data filter, and the nucleotides from the genetic sequence of the organism which are different from the nucleotides the surprisal data filter, discarding sequences of nucleotides that are the same in the genetic sequence of the organism;
• and
• reporting the status of the task to map the surprisal data filter to the uncompressed genetic sequence to the at least one master node of the cluster;
• when a worker node has reported a completion of the map job, program instructions, stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to task the worker node with a reduce job based on a specific key, comprising:
• the worker node shuffling the intermediate surprisal data between the worker node and a plurality of worker nodes of other clusters, based on the specific key;
• the worker node reducing the intermediate surprisal data to an output of surprisal data and associated metadata.