# US 11,442,828 B2 · Handling failures in distributed data system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,828 B2 - Document ID: US11442828B2 - Status: Granted - Kind: B2 - Application number: 17/262,730 - Filed: 2019-07-23 - Published: 2021-06-24 - Granted: 2022-09-13 - Earliest claimed: 2018-07-23 - Assignees: Technion Research & Development Foundation Limited - Applicants: Technion Research & Development Foundation Limited - Inventors: Oded Shmueli; Itai Shaked - Web page: https://patentgrep.com/patent/US11442828B2 ## CPC - G06F 11/2097: {maintaining the standby controller/processing unit updated (initialisation or re-synchronisation thereof G06F11/1658 and subgroups)} - G06F 11/1402: {Saving, restoring, recovering or retrying} - G06F 11/2094: {Redundant storage or storage space (G06F11/2056 takes precedence)} - G06F 16/2365: {Ensuring data consistency and integrity} - G06F 16/278: {Data partitioning, e.g. horizontal or vertical partitioning} - G06F 2201/80: Database-specific techniques - G06F 2201/84: Using snapshots, i.e. a logical point-in-time copy of the data - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method comprising: prior to performing a planned i-th data movement in a distributed data system comprising a plurality of storage sites, generating (i) a first map of current locations of data item replicas following an immediately-preceding (i−1)-th data movement, (b) a first timestamp associated with a completion of said (i−1)-th data movement, and (c) a second map of expected locations of data item replicas following said i-th data movement; implementing said i-th data movement; generating, upon performing said i-th data movement, a second timestamp associated with a completion of said i-th data movement, wherein said second timestamp is indexed as i; and if a failure occurs with respect to at least one of said storage sites during said i-th data movement, determining a completion status of said i-th data movement, based on a comparison between said indexing of said second map and said indexing of said second timestamp. ## Claims ### Claim 1 A method comprising: prior to performing a planned i-th data movement in a distributed data system comprising a plurality of storage sites, generating (i) a first map of current locations of data item replicas following an immediately-preceding (i−1)-th data movement, wherein said first map is indexed as i−1, (b) a first timestamp associated with a completion of said (i−1)-th data movement, wherein said first timestamp is indexed as i−1, and (c) a second map of expected locations of data item replicas following said i-th data movement, wherein said second map is indexed as i; implementing said i-th data movement; generating, upon performing said i-th data movement, a second timestamp associated with a completion of said i-th data movement, wherein said second timestamp is indexed as i; and if a failure occurs with respect to at least one of said storage sites during said i-th data movement, determining a completion status of said i-th data movement, based on a comparison between said indexing of said second map and said indexing of said second timestamp. ### Claim 2 (depends on 1) The method of claim 1, further comprising storing said first map, said first timestamp, and said second map, in a specified directory at each of said storage sites. ### Claim 3 (depends on 2) The method of claim 2, further comprising saving, in each of said storage sites, information regarding a location of all said specified directories. ### Claim 4 (depends on 1) The method of claim 1, wherein, when said indexing of said second timestamp is equal to said indexing of said second map, said completion status is successful. ### Claim 5 (depends on 1) The method of claim 1, wherein when said indexing of said second timestamp is lower than said indexing of said second map, said completion status is unsuccessful. ### Claim 6 (depends on 5) The method of claim 5, further comprising performing data recovery when said completion status is unsuccessful, and wherein, when said failure comprises a failure of at least one instance of a global service registry, said recovery comprises using said global service registry to initiate one or more new instances of said global service registry. ### Claim 7 (depends on 6) The method of claim 6, wherein, when said failure comprises a failure of all instances of said global service registry, said recovery comprises restoring at least some of said data items to said expected locations based, at least in part, on said second map. ### Claim 8 (depends on 7) The method of claim 7, wherein at least some of said data items are excluded from said recovery based, at least in part, when a location of each of said excluded data items corresponds to a respective expected location of said excluded data item, and wherein said correspondence is determined based, at least in part, on one of: a hashing function, and said second map. ### Claim 9 A system comprising: at least one hardware processor; and a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by said at least one hardware processor to prior to performing a planned i-th data movement in a distributed data system comprising a plurality of storage sites, generate (i) a first map of current locations of data item replicas following an immediately-preceding (i−1)-th data movement, wherein said first map is indexed as i−1, (b) a first timestamp associated with a completion of said (i−1)-th data movement, wherein said first timestamp is indexed as i−1, and (c) a second map of expected locations of data item replicas following said i-th data movement, wherein said second map is indexed as i; implement said i-th data movement; generate, upon performing said i-th data movement, a second timestamp associated with a completion of said i-th data movement, wherein said second timestamp is indexed as i; and if a failure occurs with respect to at least one of said storage sites during said i-th data movement, determine a completion status of said i-th data movement, based on a comparison between said indexing of said second map and said indexing of said second timestamp. ### Claim 10 (depends on 9) The system of claim 9, wherein said program instructions are further executable to store said first map, said first timestamp, and said second map, in a specified directory at each of said storage sites. ### Claim 11 (depends on 10) The system of claim 10, wherein said program instructions are further executable to save, in each of said storage sites, information regarding a location of all said specified directories. ### Claim 12 (depends on 9) The system of claim 9, wherein, when said indexing of said second timestamp is equal to said indexing of said second map, said completion status is successful. ### Claim 13 (depends on 9) The system of claim 9, wherein when said indexing of said second timestamp is lower than said indexing of said second map, said completion status is unsuccessful. ### Claim 14 (depends on 13) The system of claim 13, wherein said program instructions are further executable to perform data recovery when said completion status is unsuccessful, and wherein, when said failure comprises a failure of at least one instance of a global service registry, said recovery comprises using said global service registry to initiate one or more new instances of said global service registry. ### Claim 15 (depends on 14) The system of claim 14, wherein, when said failure comprises a failure of all instances of said global service registry, said recovery comprises restoring at least some of said data items to said expected locations based, at least in part, on said second map. ### Claim 16 (depends on 15) The system of claim 15, wherein at least some of said data items are excluded from said recovery based, at least in part, when a location of each of said excluded data items corresponds to a respective expected location of said excluded data item, and wherein said correspondence is determined based, at least in part, on one of: a hashing function, and said second map. ### Claim 17 A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to: prior to performing a planned i-th data movement in a distributed data system comprising a plurality of storage sites, generate (i) a first map of current locations of data item replicas following an immediately-preceding (i−1)-th data movement, wherein said first map is indexed as i−1, (b) a first timestamp associated with a completion of said (i−1)-th data movement, wherein said first timestamp is indexed as i−1, and (c) a second map of expected locations of data item replicas following said i-th data movement, wherein said second map is indexed as i; implement said i-th data movement; generate, upon performing said i-th data movement, a second timestamp associated with a completion of said i-th data movement, wherein said second timestamp is indexed as i; and if a failure occurs with respect to at least one of said storage sites during said i-th data movement, determine a completion status of said i-th data movement, based on a comparison between said indexing of said second map and said indexing of said second timestamp. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein said program instructions are further executable to store said first map, said first timestamp, and said second map, in a specified directory at each of said storage sites. ### Claim 19 (depends on 18) The computer program product of claim 18, wherein said program instructions are further executable to save, in each of said storage sites, information regarding a location of all said specified directories. ### Claim 20 (depends on 17) The computer program product of claim 17, wherein, when said indexing of said second timestamp is equal to said indexing of said second map, said completion status is successful, and wherein when said indexing of said second timestamp is lower than said indexing of said second map, said completion status is unsuccessful.