# US 11,030,198 B2 · Reducing resource consumption of a similarity index in data deduplication > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,030,198 B2 - Document ID: US11030198B2 - Status: Granted - Kind: B2 - Application number: 15/953,755 - Filed: 2018-04-16 - Published: 2018-08-16 - Granted: 2021-06-08 - Earliest claimed: 2015-11-25 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Lior Aronovich - Web page: https://patentgrep.com/patent/US11030198B2 ## CPC - G06F 16/24556: {Aggregation; Duplicate elimination} - G06F 16/2272: {Management thereof} - G06F 16/2365: {Ensuring data consistency and integrity} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Embodiments for reducing resource consumption of a similarity index in data deduplication by a processor. Only a latest generation of repository data is represented in the similarity index. Implicit deletion is applied in the similarity index. A subset of bytes of a full representative value is maintained in a similarity index entry. A respective one of a plurality of similarity units is deduplicated with a corresponding similarity unit of a previous snapshot, and a deduplication coverage thereof is examined. If a deduplication coverage threshold is not met, a similarity search is conducted and the respective one of the similarity units is deduplicated with a found similarity unit residing in the similarity index. ## Claims ### Claim 1 A method for reducing resource consumption of a similarity index in a data deduplication system by a processor, comprising: representing in the similarity index only a latest generation of repository data, wherein a single latest entry is maintained for a representative value in the similarity index; applying implicit deletion in the similarity index such that a similarly index entry is not removed or overwritten until a change with associated data of the similarity index entry is detected; maintaining a subset of bytes of the representative value in the similarity index entry to reduce an input/output (I/O) load on the similarity index of the data deduplication system; deduplicating a respective one of a plurality of similarity units with a corresponding similarity unit of a previous snapshot; and examining a deduplication coverage; wherein if a deduplication coverage threshold is not met, a similarity search is conducted and the respective one of the plurality of similarity units is deduplicated with a unified set of digest matches combined from the corresponding similarity unit of the previous snapshot and a found similarity unit residing in the similarity index. ### Claim 2 (depends on 1) The method of claim 1, further including designating the similarity index entry for removal if at least one of: snapshot data corresponding to the similarity index entry is deleted, and new snapshot data is processed having new representative values of a new snapshot. ### Claim 3 (depends on 2) The method of claim 2, further including processing the similarity index entry for removal by at least one of: updating the new snapshot data, and identifying the similarity index entry for removal within a search operation and removing the similarity index entry for removal. ### Claim 4 (depends on 3) The method of claim 3, further including updating the similarity index entry designated to be removed by a new similarity index entry that has a same representative value. ### Claim 5 (depends on 1) The method of claim 1, further including configuring the similarity index entry to include a storage reference to digests of the respective one of the plurality of similarity units to which the representative value corresponds to. ### Claim 6 (depends on 5) The method of claim 5, further including storing remaining bytes of the representative value within the digests of the respective one of the plurality of similarity units. ### Claim 7 A system for reducing resource consumption of a similarity index in performing data deduplication, comprising: a processor, operable to perform the data deduplication, wherein the processor: represents in the similarity index only a latest generation of repository data, wherein a single latest entry is maintained for a representative value in the similarity index, applies implicit deletion in the similarity index such that a similarly index entry is not removed or overwritten until a change with associated data of the similarity index entry is detected, maintains a subset of bytes of the representative value in the similarity index entry to reduce an input/output (I/O) load on the similarity index of the system, deduplicates a respective one of a plurality of similarity units with a corresponding similarity unit of a previous snapshot, and examines a deduplication coverage; wherein if a deduplication coverage threshold is not met, a similarity search is conducted and the respective one of the plurality of similarity units is deduplicated with a unified set of digest matches combined from the corresponding similarity unit of the previous snapshot and a found similarity unit residing in the similarity index. ### Claim 8 (depends on 7) The system of claim 7, wherein the processor designates the similarity index entry for removal if at least one of: snapshot data corresponding to the similarity index entry is deleted, and new snapshot data is processed having new representative values of a new snapshot. ### Claim 9 (depends on 8) The system of claim 8, wherein the processor processes the similarity index entry for removal by at least one of: updating the new snapshot data, and identifying the similarity index entry for removal within a search operation and removing the similarity index entry for removal. ### Claim 10 (depends on 9) The system of claim 9, wherein the processor updates the similarity index entry designated to be removed by a new similarity index entry that has a same representative value. ### Claim 11 (depends on 7) The system of claim 7, wherein the processor configures the similarity index entry to include a storage reference to digests of the respective one of the plurality of similarity units to which the representative value corresponds to. ### Claim 12 (depends on 11) The system of claim 11, wherein the processor stores remaining bytes of the representative value within the digests of the respective one of the plurality of similarity units. ### Claim 13 A computer program product for reducing resource consumption of a similarity index in a data deduplication system by a processor, the computer program product comprising a computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising: an executable portion that represents in the similarity index only a latest generation of repository data, wherein a single latest entry is maintained for a representative value in the similarity index; an executable portion that applies implicit deletion in the similarity index such that a similarly index entry is not removed or overwritten until a change with associated data of the similarity index entry is detected; an executable portion that maintains a subset of bytes of the representative value in the similarity index entry to reduce an input/output (I/O) load on the similarity index of the data deduplication system; an executable portion that deduplicates a respective one of a plurality of similarity units with a corresponding similarity unit of a previous snapshot; and an executable portion that examines a deduplication coverage; wherein if a deduplication coverage threshold is not met, a similarity search is conducted and the respective one of the plurality of similarity units is deduplicated with a unified set of digest matches combined from the corresponding similarity unit of the previous snapshot and a found similarity unit residing in the similarity index. ### Claim 14 (depends on 13) The computer program product of claim 13, further including an executable portion that designates the similarity index entry for removal if at least one of: snapshot data corresponding to the similarity index entry is deleted, and new snapshot data is processed having new representative values of a new snapshot. ### Claim 15 (depends on 14) The computer program product of claim 14, further including an executable portion that processes the similarity index entry for removal by at least one of: updating the new snapshot data, and identifying the similarity index entry for removal within a search operation and removing the similarity index entry for removal. ### Claim 16 (depends on 15) The computer program product of claim 15, further including an executable portion that updates the similarity index entry designated to be removed by a new similarity index entry that has a same representative value. ### Claim 17 (depends on 13) The computer program product of claim 13, further including an executable portion that configures the similarity index entry to include a storage reference to digests of the respective one of the plurality of similarity units to which the representative value corresponds to. ### Claim 18 (depends on 17) The computer program product of claim 17, further including an executable portion that stores remaining bytes of the representative value within the digests of the respective one of the plurality of similarity units.