# US 12,222,913 B2 · Efficient method to optimize distributed segment processing mechanism in dedupe systems by leveraging the locality principle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,222,913 B2 - Document ID: US12222913B2 - Status: Granted - Kind: B2 - Application number: 18/525,344 - Filed: 2023-11-30 - Published: 2024-04-04 - Granted: 2025-02-11 - Earliest claimed: 2021-03-03 - Assignees: EMC IP Holding Company LLC - Applicants: EMC IP Holding Company LLC - Inventors: Kalyan C. Gunda; Jagannathdas Rath - Web page: https://patentgrep.com/patent/US12222913B2 ## CPC - G06F 11/1453: {using de-duplication of the data} - G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors - G06F 16/2379: {Updates performed during online database operations; commit processing} - G06F 2201/83: the solution involving signatures - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract One example method includes receiving at a dedupe system, from a client, a request that comprises a set of fingerprints, where each fingerprint in the set corresponds to a particular data segment, filtering, at the dedupe system, the set of fingerprints into a set of unique fingerprints and a set of non-unique fingerprints, reading, at the dedupe system, from a container where copies of the non-unique fingerprints are stored, an additional set of non-unique fingerprints, sending, from the dedupe system to the client, a single response that comprises both the set of unique fingerprints and the additional set of non-unique fingerprints, and receiving from the client, at the dedupe system, data segments that respectively correspond to the unique fingerprints in the set of unique fingerprints, but no data segments corresponding to the non-unique fingerprints in the set of non-unique fingerprints are received by the dedupe system from the client. ## Claims ### Claim 1 A method, comprising: sending, by a client to a dedupe system, a set of fingerprints for filtering by the dedupe system; receiving, by the client from the dedupe system, fingerprints of the set of fingerprints that were identified by the dedupe system as unique fingerprints, and additional non-unique fingerprints from a set of containers of the dedupe system; sending, by the client to the dedupe system, only unique segments for writing by the dedupe system, and the unique segments form a set; receiving, by the client system, data from a backup dataset, and segmenting and fingerprinting the data from the backup dataset to create a next set of fingerprints; filtering, by the client system, fingerprints in the next set of fingerprints, using additional existing fingerprints received earlier from the dedupe system, so as to identify unique segments of the next set; and sending, by the client system to the dedupe system, a segment tree of the unique segments of the next set. ### Claim 2 (depends on 1) The method as recited in claim 1, wherein any fingerprints identified, during the filtering, as matching fingerprints already present in a local in-memory list, are not sent by the client system to the dedupe system. ### Claim 3 (depends on 2) The method as recited in claim 2, wherein the local in-memory list includes one or more non-unique fingerprints previously sent by the client system to the dedupe system. ### Claim 4 (depends on 1) The method as recited in claim 1, wherein the segment tree is usable by the dedupe system to build file metadata for a file. ### Claim 5 (depends on 1) The method as recited in claim 1, wherein an amount of the unique fingerprints sent by the dedupe system to the client is specified by the client. ### Claim 6 (depends on 1) The method as recited in claim 1, wherein an amount of the unique fingerprints sent by the dedupe system to the client is a function of a value of an adaptive match rate that is determined by the client. ### Claim 7 (depends on 1) The method as recited in claim 1, wherein the unique fingerprints, and the additional non-unique fingerprints from the set of containers of the dedupe system, are both received by the client in response to a single RPC (remote procedure call) issued by the client to the dedupe system. ### Claim 8 (depends on 1) The method as recited in claim 1, the unique fingerprints, and the additional non-unique fingerprints received by the client from the dedupe system, are written by the client to an in-memory list locally maintained at the client. ### Claim 9 (depends on 1) The method as recited in claim 1, wherein the segment tree is sent at an end of a data ingest process at the dedupe system. ### Claim 10 (depends on 1) The method as recited in claim 1, wherein the unique segments are created by the client. ### Claim 11 A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising: sending, by a client to a dedupe system, a set of fingerprints for filtering by the dedupe system; receiving, by the client from the dedupe system, fingerprints of the set of fingerprints that were identified by the dedupe system as unique fingerprints, and additional non-unique fingerprints from a set of containers of the dedupe system; sending, by the client to the dedupe system, only unique segments for writing by the dedupe system, and the unique segments form a set; receiving, by the client system, data from a backup dataset, and segmenting and fingerprinting the data from the backup dataset to create a next set of fingerprints; filtering, by the client system, fingerprints in the next set of fingerprints, using additional existing fingerprints received earlier from the dedupe system, so as to identify unique segments of the next set; and sending, by the client system to the dedupe system, a segment tree of the unique segments of the next set. ### Claim 12 (depends on 11) The non-transitory storage medium as recited in claim 11, wherein any fingerprints identified, during the filtering, as matching fingerprints already present in a local in-memory list, are not sent by the client system to the dedupe system. ### Claim 13 (depends on 2) The non-transitory storage medium as recited in claim 2, wherein the local in-memory list includes one or more non-unique fingerprints previously sent by the client system to the dedupe system. ### Claim 14 (depends on 11) The non-transitory storage medium as recited in claim 11, wherein the segment tree is usable by the dedupe system to build file metadata for a file. ### Claim 15 (depends on 11) The non-transitory storage medium as recited in claim 11, wherein an amount of the unique fingerprints sent by the dedupe system to the client is specified by the client. ### Claim 16 (depends on 11) The non-transitory storage medium as recited in claim 11, wherein an amount of the unique fingerprints sent by the dedupe system to the client is a function of a value of an adaptive match rate that is determined by the client. ### Claim 17 (depends on 11) The non-transitory storage medium as recited in claim 11, wherein the unique fingerprints, and the additional non-unique fingerprints from the set of containers of the dedupe system, are both received by the client in response to a single RPC (remote procedure call) issued by the client to the dedupe system. ### Claim 18 (depends on 11) The non-transitory storage medium as recited in claim 11, the unique fingerprints, and the additional non-unique fingerprints received by the client from the dedupe system, are written by the client to an in-memory list locally maintained at the client. ### Claim 19 (depends on 11) The non-transitory storage medium as recited in claim 11, wherein the segment tree is sent at an end of a data ingest process at the dedupe system. ### Claim 20 (depends on 11) The non-transitory storage medium as recited in claim 11, wherein the unique segments are created by the client.