# US 2016/0366226 A1 · Deduplication in a highly-distributed shared topology with direct-memory-access capable interconnect > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2016/0366226 A1 - Document ID: US20160366226A1 - Status: Published application - Kind: A1 - Application number: 15/168,055 - Filed: 2016-05-29 - Published: 2016-12-15 - Earliest claimed: 2015-06-11 - Assignees: E8 Storage Systems Ltd. - Applicants: E8 Storage Systems Ltd. - Inventors: Alex Friedman; Barak Pinhas; Evgeny Budilovsky - Web page: https://patentgrep.com/patent/US20160366226A1 ## CPC - G06F 16/2255: {Hash tables} - G06F 3/0608: {Saving storage space on storage systems} - G06F 3/061: {Improving I/O performance} - G06F 3/0641: {De-duplication techniques} - G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method for data storage includes, in a system that includes multiple servers, multiple multi-queue storage devices and at least one storage controller that communicate over a network, storing data blocks by the servers on the storage devices. A shared data structure, which is accessible to the servers using remote direct memory access and which maps hash values calculated over the data blocks to respective storage locations of the data blocks on the storage devices, is maintained. The data blocks stored on the storage device are deduplicated, by looking-up the shared data structure by the servers without executing code on a processor of the storage controller. ## Claims ### Claim 1 A method for data storage, comprising: in a system that comprises multiple servers, multiple multi-queue storage devices and at least one storage controller that communicate over a network, storing data blocks by the servers on the storage devices; maintaining a shared data structure, which is accessible to the servers using remote direct memory access and which maps hash values calculated over the data blocks to respective storage locations of the data blocks on the storage devices; and deduplicating the data blocks stored on the storage device, by looking-up the shared data structure by the servers without executing code on a processor of the storage controller. ### Claim 2 (depends on 1) The method according to claim 1, wherein the shared data structure comprises a Key-Value (KV) store. ### Claim 3 (depends on 1) The method according to claim 1, wherein deduplicating the data blocks comprises, in a given server that prepares to store a given data block, checking whether a hash value of the given data block already exists in the shared data structure, and if the hash value exists, refraining from storing the given data block on the storage devices. ### Claim 4 (depends on 3) The method according to claim 3, wherein deduplicating the data blocks further comprises, if the hash value exists, mapping a logical address of the given data block to a storage location to which the existing hash value is mapped in the shared data structure. ### Claim 5 (depends on 3) The method according to claim 3, wherein deduplicating the data blocks further comprises, if the hash value exists, updating a shared reference count associated with the existing hash value. ### Claim 6 (depends on 3) The method according to claim 3, wherein deduplicating the data blocks further comprises, if the hash value exists, updating a shared bit store to indicate that the given data block is referenced more than once. ### Claim 7 (depends on 1) The method according to claim 1, wherein deduplicating the data blocks further comprises running a background garbage-collection process that identifies and releases unreferenced data blocks. ### Claim 8 (depends on 1) The method according to claim 1, further comprising running by the storage controller a background deduplication process that deduplicates data blocks already stored on the storage devices, wherein deduplication operations performed by the background deduplication process are seen atomically by the servers. ### Claim 9 (depends on 1) The method according to claim 1, wherein deduplicating the data blocks comprises choosing whether or not to deduplicate a given data block, depending on a logical volume to which the given data block belongs. ### Claim 10 (depends on 1) The method according to claim 1, wherein deduplicating the data blocks comprises calculating the hash values by one or more Network Interface Controllers (NICs) of the servers, the storage devices or the storage controller. ### Claim 11 A system for data storage, comprising: at least one storage controller; and multiple servers, which are configured to communicate over a network with the storage controller and with multiple multi-queue storage devices, to store data blocks on the storage devices, to access, using remote direct memory access, a shared data structure that maps hash values calculated over the data blocks to respective storage locations of the data blocks on the storage devices, and to deduplicate the data blocks stored on the storage device, by looking-up the shared data structure by the servers without executing code on a processor of the storage controller. ### Claim 12 (depends on 11) The system according to claim 11, wherein the shared data structure comprises a Key-Value (KV) store. ### Claim 13 (depends on 11) The system according to claim 11, wherein a given server is configured to deduplicate the data blocks by, upon preparing to store a given data block, checking whether a hash value of the given data block already exists in the shared data structure, and if the hash value exists, refraining from storing the given data block on the storage devices. ### Claim 14 (depends on 13) The system according to claim 13, wherein the given server is further configured, if the hash value exists, to map a logical address of the given data block to a storage location to which the existing hash value is mapped in the shared data structure. ### Claim 15 (depends on 13) The system according to claim 13, wherein the given server is further configured, if the hash value exists, to update a shared reference count associated with the existing hash value. ### Claim 16 (depends on 13) The system according to claim 13, wherein the given server is further configured, if the hash value exists, to update a shared bit store to indicate that the given data block is referenced more than once. ### Claim 17 (depends on 11) The system according to claim 11, wherein the storage controller is configured to run a background garbage-collection process that identifies and releases unreferenced data blocks. ### Claim 18 (depends on 11) The system according to claim 11, wherein the storage controller is configured to run a background deduplication process that deduplicates data blocks already stored on the storage devices, wherein deduplication operations performed by the background deduplication process are seen atomically by the servers. ### Claim 19 (depends on 11) The system according to claim 11, wherein the servers are configured to choose whether or not to deduplicate a given data block, depending on a logical volume to which the given data block belongs. ### Claim 20 (depends on 11) The system according to claim 11, wherein one or more of: the storage controller and one or more of the servers comprise one or more Network Interface Controllers (NICs), which are configured to communicate over the network and also to calculate the hash values. ### Claim 21 A server, comprising: a Network Interface Controller (NIC) configured to connect to a network; and a processor, which is configured to communicate over the network with at least one storage controller and with multiple multi-queue storage devices, to store data blocks on the storage devices, to access, using remote direct memory access, a shared data structure that maps hash values calculated over the data blocks to respective storage locations of the data blocks on the storage devices, and to deduplicate the data blocks stored on the storage device, by looking-up the shared data structure by the servers without executing code on any processor of the storage controller. ### Claim 22 (depends on 21) The server according to claim 21, wherein the processor is configured, upon preparing to store a given data block, to check whether a hash value of the given data block already exists in the shared data structure, and if the hash value exists, to refrain from storing the given data block on the storage devices.