# US 10,310,938 B2 · Data deduplication with reduced hash computations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,310,938 B2 - Document ID: US10310938B2 - Status: Granted - Kind: B2 - Application number: 15/823,891 - Filed: 2017-11-28 - Published: 2018-03-22 - Granted: 2019-06-04 - Earliest claimed: 2016-04-29 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Timothy J. Fisher; Nikolas Ioannou; Thomas Parnell; Roman A. Pletka; Sasa Tomic - Web page: https://patentgrep.com/patent/US10310938B2 ## CPC - G06F 11/1004: {to protect a block of data words, e.g. CRC or checksum (G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00)} - G06F 3/0608: {Saving storage space on storage systems} - G06F 3/0641: {De-duplication techniques} - G06F 3/0644: {Management of space entities, e.g. partitions, extents, pools} - G06F 3/0661: {Format or protocol conversion arrangements} - G06F 3/0688: {Non-volatile semiconductor memory arrays} - G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Techniques for data deduplication in a data storage system include comparing a first attribute of a received data page to first attributes of one or more stored data pages. In response to the first attribute matching one of the first attributes, a second attribute of the received data page is compared to second attributes of the one or more data pages. In response to the second attribute of the received data page matching one of the second attributes, a fingerprint of the received data page is compared to fingerprints of the one or more data pages. In response to the fingerprint of the received data page matching one of the fingerprints, the received data page is discarded and replaced with a reference to the corresponding data page already stored in the storage system. In response to first attribute, the second attribute, or the fingerprint of the received data page not matching, the received data page is stored. ## Claims ### Claim 1 A method of data deduplication for a data storage system, comprising: comparing, by a controller, a first attribute of a received data page to one or more corresponding first attributes of one or more data pages stored in the storage system; in response to the first attribute of the received data page not being the same as one or more of the first attributes, storing, by the controller, the received data page in the storage system; in response to the first attribute being the same as one or more of the first attributes, comparing, by the controller, a second attribute of the received data page to one or more corresponding second attributes of the one or more data pages stored in the storage system; in response to the second attribute of the received data page not being the same as one or more of the second attributes, storing, by the controller, the received data page in the storage system; in response to the second attribute of the received data page being the same as one or more of the second attributes, comparing, by the controller, a fingerprint of the received data page to one or more fingerprints of the one or more data pages stored in the storage system, wherein generating the first and second attributes of the received data page is less computationally expensive than generating the fingerprint of the received data page; in response to the fingerprint of the received data page not being the same as one or more of the fingerprints, storing, by the controller, the received data page in the storage system; and in response to the fingerprint of the received data page being the same as one or more of the fingerprints, replacing, by the controller, the received data page with a reference to a corresponding data page included in the one or more data pages stored in the storage system and discarding the received data page, wherein the first attribute corresponds to a cyclic redundancy check (CRC) value, the second attribute corresponds to a compressed page size, and the method further comprises: generating the CRC value for the received data page; and in response to the CRC value of the received data page being the same as a CRC value of one of the data pages stored in the storage system, compressing the received data page to generate the compressed page size. ### Claim 2 (depends on 1) The method of claim 1, further comprising: generating the fingerprint for the received data page. ### Claim 3 (depends on 1) The method of claim 1, further comprising: requesting, by an interface node of the storage system, scanning on a flash card of the storage system for one or more compressed page sizes stored in a table that associates compressed page sizes with logical addresses of stored data pages; and receiving, by the interface node, a result of the scanning from the flash card. ### Claim 4 (depends on 1) The method of claim 1, further comprising: requesting, by an interface node of the storage system, scanning on a flash card of the storage system for one or more cyclic redundancy check (CRC) values; and receiving, by the interface node, a result of the scanning from the flash card, wherein the CRC values are stored in a NAND flash memory system of the flash card. ### Claim 5 (depends on 4) The method of claim 4, further comprising: adjusting the CRC values to remove information not related to a logical data page. ### Claim 6 (depends on 1) The method of claim 1, wherein the received data page is read from a flash card by a background deduplication process. ### Claim 7 A data storage system, comprising: an interface node coupled to one or more data storage devices of the data storage system, wherein the interface node is configured to: compare a first attribute of a received data page to one or more corresponding first attributes of one or more data pages stored in the storage system; in response to the first attribute of the received data page not being the same as one or more of the first attributes, store the received data page in the storage system; in response to the first attribute being the same as one or more of the first attributes, compare a second attribute of the received data page to one or more corresponding second attributes of the one or more data pages stored in the storage system; in response to the second attribute of the received data page not being the same as one or more of the second attributes, store the received data page in the storage system; in response to the second attribute of the received data page being the same as one or more of the second attributes, compare a fingerprint of the received data page to one or more fingerprints of the one or more data pages stored in the storage system, wherein generating the first and second attributes of the received data page is less computationally expensive than generating the fingerprint of the received data page; in response to the fingerprint of the received data page not being the same as one or more of the fingerprints, store the received data page in the storage system; and in response to the fingerprint of the received data page being the same as one or more of the fingerprints, replace the received data page with a reference to a corresponding data page included in the one or more data pages stored in the storage system and discard the received data page, wherein the first attribute corresponds to a cyclic redundancy check (CRC) value, the second attribute corresponds to a compressed page size, and the interface node is further configured to: generate the CRC value for the received data page; and in response to the CRC value of the received data page being the same as a CRC value of one of the data pages stored in the storage system, compress the received data page to generate the compressed page size. ### Claim 8 (depends on 7) The data storage system of claim 7, and the interface node is further configured to: generate the fingerprint for the received data page. ### Claim 9 (depends on 7) The data storage system of claim 7, wherein the interface node is further configured to: request scanning on a flash card of the storage system for one or more compressed page sizes stored in a table that associates compressed page sizes with logical addresses of stored data pages; and receive a result of the scanning from the flash card. ### Claim 10 (depends on 7) The data storage system of claim 7, wherein the interface node is further configured to: request scanning on a flash card of the storage system for one or more cyclic redundancy check (CRC) values; and receive a result of the scanning from the flash card, wherein the CRC values are stored in a NAND flash memory system of the flash card. ### Claim 11 A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a controller, to cause the controller to: compare a first attribute of a received data page to one or more corresponding first attributes of one or more data pages; in response to the first attribute of the received data page not being the same as one or more of the first attributes, store the received data page; in response to the first attribute being the same as one or more of the first attributes, compare a second attribute of the received data page to one or more corresponding second attributes of the one or more data pages stored in the storage system; in response to the second attribute of the received data page not being the same as one or more of the second attributes, store the received data page in the storage system; in response to the second attribute of the received data page being the same as one or more of the second attributes, compare a fingerprint of the received data page to one or more fingerprints of the one or more data pages stored in the storage system, wherein generating the first and second attributes of the received data page is less computationally expensive than generating the fingerprint of the received data page; in response to the fingerprint of the received data page not being the same as one or more of the fingerprints, store the received data page; and in response to the fingerprint of the received data page being the same as one or more of the fingerprints, replace the received data page with a reference to a corresponding data page included in the one or more data pages and discard the received data page, wherein the first attribute corresponds to a cyclic redundancy check (CRC) value, the second attribute corresponds to a compressed page size, and the program instructions further cause the controller to: generate the CRC value for the received data page; and in response to the CRC value of the received data page being the same as a CRC value of one of the data pages stored in the storage system, compress the received data page to generate the compressed page size. ### Claim 12 (depends on 11) The program product of claim 11, wherein the program instructions further cause the controller to: generate the fingerprint for the received data page. ### Claim 13 (depends on 11) The program product of claim 11, wherein the program instructions further cause the controller to: request scanning on a flash card of the storage system for one or more compressed page sizes stored in a table that associates compressed page sizes with logical addresses of stored data pages; and receive a result of the scanning from the flash card. ### Claim 14 (depends on 11) The program product of claim 11, wherein the program instructions further cause the controller to: request scanning on a flash card of the storage system for one or more cyclic redundancy check (CRC) values; and receive a result of the scanning from the flash card, wherein the CRC values are stored in a NAND flash memory system of the flash card.