Storage and communication de-duplication

#The amount of data that even casual users generate and interact with is ever increasing. For example, a user may interact with a social network service, store photos, communicate the photos with other users, read and respond to emails and instant messages, view videos, and so on. Consequently, the resources used to support this interaction (e.g., data storage and communication) may become burdened by this amount of data. Traditional techniques that were utilized to alleviate at least some of this strain, however, were often disjointed and therefore could also consume significant amounts of resources and limit the sophistication of the techniques that could be employed.

#Storage and communication de-duplication are described. In one or more implementations, a system comprises one or more modules that are implemented at least partially in hardware, the one or more modules configured to utilize one or more algorithms to calculate hashes of chunks of data, the hashes used to replace the chunks in the data for storage locally in the system as well as to communicate the hashes in response to a request received via a network for the data to avoid communicating at least one of the chunks of the data via the network.

#In one or more implementations, a sending device replaces one or more chunks of data with the one or more identifiers, stores the one or more chunks and the one or more identifiers, and stores the data as having the one or more identifiers. Responsive to a request to communicate the data via a network, a communication is formed to communicate via the network, the communication having the one or more identifiers of the one or more chunks from the data. The one or more identifiers are configured for use by a receiving device to identify whether the one or more chunks are available locally at the receiving device.

#The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different instances in the description and the figures may indicate similar or identical items.

#FIG. 1 is an illustration of an environment in an example implementation that is operable to perform storage and communication de-duplication techniques.

#FIG. 2 is an illustration of a system in an example implementation showing generation of chunks and identifiers for use in storage of a file.

#Overview

#Previous techniques used to increase efficiency in the utilization of file storage and file communication often involved dedicated devices. Further, these techniques were often fractured and thus limited the sophistication of algorithms that could be employed in traditional scenarios.

#Storage and communication de-duplication techniques are described. In one or more implementations, storage and communication de-duplication are unified to reduce redundancy and increase efficiency of data storage and communication. These techniques may be used to chunk data and calculate hashes of the chunks for storage. The chunks may then be stored once and the hashes used to identify the chunks in the original data. Therefore, this data could be reconstructed using the hashes and the chunks when access is desired.