# US 8,572,137 B2 · Data de-duplication in a distributed network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,572,137 B2 - Document ID: US8572137B2 - Status: Granted - Kind: B2 - Application number: 12/555,703 - Filed: 2009-09-08 - Published: 2011-03-10 - Granted: 2013-10-29 - Earliest claimed: 2009-09-08 - Assignees: International Business Machines Corporation - Applicants: Shachar Fienblit; Itzhack Goldberg; Evgeny Schmeilin; Aviad Zlotnick - Inventors: Shachar Fienblit; Itzhack Goldberg; Evgeny Schmeilin; Aviad Zlotnick - Web page: https://patentgrep.com/patent/US8572137B2 ## CPC - G06F 16/1748: {De-duplication implemented within the file system, e.g. based on file segments (de-duplication techniques in storage systems for the management of data blocks G06F3/0641)} - G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computer-implemented method for efficient data storage is provided. A first storage medium associates data stored on one or more data storage media with a unique identification value (ID) for the purpose of determining de-duplication status of the data. In response to receiving a request to read the data from a logical address, the first storage medium retrieves the data from a second storage medium based on the unique ID. In response to receiving a request to write the data to a logical address, the one or more data storage media store at least one copy of the data based on the de-duplication status of the data. ## Claims ### Claim 1 A computer-implemented method for de-duplication of data in a distributed network, the method comprising: receiving, by a first de-duplication manager (DDM) in the distributed network, at least a unique identification (ID) of the data and a network address of a first storage medium in which the data is stored; locating one or more storage media in the distributed network in which the data is stored using an association of the unique ID of the data and one or more physical addresses where the data is stored, wherein a logical address of the data is associated with network addresses of the one or more storage media; determining, via the association, if there is more than a predetermined threshold number of copies of the data; and if there is more than the predetermined threshold number of copies of the data: selecting one or more copies of the data for removal, and removing the selected one or more copies of the data from a second storage medium selected from among the one or more storage media, wherein selecting the one or more copies comprises selecting the one or more copies of the data that are furthest from a client that frequently accesses the data. ### Claim 2 (depends on 1) The method of claim 1, wherein the second storage medium is selected according to a dynamic evaluation based on at least one of affinity, frequency of use, price of storage, or estimated risk of failure. ### Claim 3 (depends on 1) The method of claim 1, wherein the removing comprises providing the unique ID of the data to the second storage medium and requesting the second storage medium to remove the data. ### Claim 4 (depends on 1) The method of claim 1, wherein the second storage medium uses the unique ID of the data to locate the data in memory and removes the data. ### Claim 5 (depends on 1) The method of claim 1, further comprising adding a first association comprising at least the unique ID of the data and the network address of the first storage medium to a first database. ### Claim 6 (depends on 5) The method of claim 5, further comprising removing a second association comprising at least the unique ID of the data and the network address of the second storage medium from the first database. ### Claim 7 (depends on 1) The method of claim 1, wherein the second storage medium removes an association comprising the unique ID of the data and a physical address of the data in the second storage medium is removed from a second database. ### Claim 8 (depends on 7) The method of claim 7, wherein the second storage medium removes an association comprising the unique ID of the data, a logical address of the data, and the network address of the second storage medium from the second database. ### Claim 9 (depends on 1) The method of claim 1, further comprising designating the data for background processing by a second DDM in the distributed network. ### Claim 10 A system for de-duplication of data in a distributed network, comprising: a processor; and a memory coupled to the processor, wherein the memory stores computer code that, when executed by the processor, cause the processor to: receive, by a first de-duplication manager (DDM) in the distributed network, at least a unique identification (ID) of the data and a network address of a first storage medium in which the data is stored, locate one or more storage media in the distributed network in which the data is stored using an association of the unique ID of the data and one or more physical addresses where the data is stored, wherein a logical address of the data is associated with network addresses of the one or more storage media; determine, via the association, if there is more than a predetermined threshold number of copies of the data, and if there is more than the predetermined threshold number of copies of the data: select one or more copies of the data for removal, and remove the selected one or more copies of the data from a second storage medium selected from among the one or more storage media, wherein selecting the one or more copies comprises selecting the one or more copies of the data that are furthest from a client that frequently accesses the data. ### Claim 11 (depends on 10) The system of claim 10, wherein the second storage medium is selected according to a dynamic evaluation based on at least one of affinity, frequency of use, price of storage, or estimated risk of failure. ### Claim 12 (depends on 10) The system of claim 10, wherein, when removing the selected one or more copies of the data, the processor is configured to provide the unique ID of the data to the second storage medium and requesting the second storage medium to remove the data. ### Claim 13 (depends on 10) The system of claim 10, wherein the second storage medium uses the unique ID of the data to locate the data in the memory and remove the data. ### Claim 14 (depends on 10) The system of claim 10, wherein the processor is further configured to add a first association comprising at least the unique ID of the data and the network address of the first storage medium to a first database. ### Claim 15 (depends on 14) The system of claim 14, wherein the processor is further configured to remove a second association comprising at least the unique ID of the data and the network address of the second storage medium from the first database. ### Claim 16 (depends on 10) The system of claim 10, wherein the second storage medium removes an association comprising the unique ID of the data and a physical address of the data in the second storage medium is removed from a second database. ### Claim 17 (depends on 16) The system of claim 16, wherein the second storage medium removes an association comprising the unique ID of the data, a logical address of the data, and the network address of the second storage medium from the second database. ### Claim 18 (depends on 10) The system of claim 10, wherein the processor is further configured to designate the data for background processing by a second DDM in the distributed network. ### Claim 19 A physical computer storage memory comprising a computer program product method for de-duplication of data in a distributed network, the physical computer storage memory comprising: computer code for receiving, by a first de-duplication manager (DDM) in the distributed network, at least a unique identification (ID) of the data and a network address of a first storage medium in which the data is stored; computer code for locating one or more storage media in the distributed network in which the data is stored using an association of the unique ID of the data and one or more physical addresses where the data is stored, wherein a logical address of the data is associated with network addresses of the one or more storage media; computer code for determining, via the association, if there is more than a predetermined threshold number of copies of the data; and if there is more than the predetermined threshold number of copies of the data: computer code for selecting one or more copies of the data for removal, and computer code for removing the selected one or more copies of the data from a second storage medium selected from among the one or more storage media, wherein selecting the one or more copies comprises selecting the one or more copies of the data that are furthest from a client that frequently accesses the data. ### Claim 20 (depends on 19) The physical computer storage memory of claim 19, wherein the second storage medium is selected according to a dynamic evaluation based on at least one of affinity, frequency of use, price of storage, or estimated risk of failure. ### Claim 21 (depends on 19) The physical computer storage memory of claim 19, wherein the removing comprises providing the unique ID of the data to the second storage medium and requesting the second storage medium to remove the data. ### Claim 22 (depends on 19) The physical computer storage memory of claim 19, wherein the second storage medium uses the unique ID of the data to locate the data in memory and removes the data. ### Claim 23 (depends on 19) The physical computer storage memory of claim 19, further comprising computer code for adding a first association comprising at least the unique ID of the data and the network address of the first storage medium to a first database. ### Claim 24 (depends on 19) The physical computer storage memory of claim 19, wherein the second storage medium removes an association comprising the unique ID of the data and a physical address of the data in the second storage medium is removed from a second database.