# US 10,310,943 B2 · Distributed data object management system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,310,943 B2 - Document ID: US10310943B2 - Status: Granted - Kind: B2 - Application number: 15/626,070 - Filed: 2017-06-16 - Published: 2018-12-20 - Granted: 2019-06-04 - Earliest claimed: 2017-06-16 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Cheng Huang; Jin Li; Aaron William Ogus; Douglas W. Phillips; Yu Lin Chen; Shuai Mu; Jinyang Li - Web page: https://patentgrep.com/patent/US10310943B2 ## CPC - G06F 11/1084: {Degraded mode, e.g. caused by single or multiple storage removals or disk failures} - G06F 3/06: Digital input from, or digital output to, record carriers {, e.g. RAID, emulated record carriers or networked record carriers} - G06F 3/0613: {in relation to throughput} - G06F 3/0635: {by changing the path, e.g. traffic rerouting, path reconfiguration} - G06F 3/064: {Management of blocks} - G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]} - G06F 11/1076: {Parity data used in redundant arrays of independent storages, e.g. in RAID systems} - G06F 11/1658: {Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit} - G06F 11/1662: {the resynchronized component or unit being a persistent storage device (re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008)} - G06F 11/1666: {where the redundant component is memory or memory area} - G06F 11/2094: {Redundant storage or storage space (G06F11/2056 takes precedence)} - G06F 11/2097: {maintaining the standby controller/processing unit updated (initialisation or re-synchronisation thereof G06F11/1658 and subgroups)} - G06F 12/0253: {Garbage collection, i.e. reclamation of unreferenced memory} - H03M 13/154: {Error and erasure correction, e.g. by using the error and erasure locator or Forney polynomial} - H03M 13/373: {with erasure correction and erasure determination, e.g. for packet loss recovery or setting of erasures for the decoding of Reed-Solomon codes} - H03M 13/3761: {using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes} - G06F 2201/805: Real-time - G06F 2212/1044: Space efficiency improvement - H03M 13/1515: {Reed-Solomon codes} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H03M: CODING; DECODING; CODE CONVERSION IN GENERAL (using fluidic means F15C4/00; optical analogue/digital converters G02F7/00; coding, decoding or code conversion, specially adapted for particular applications, see the relevant subclasses, e.g. G01D, G01R, G06F, G06T, G09G, G10L, G11B, G11C, H04B, H04L, H04M, H04N; ciphering or deciphering for cryptography or other purposes involving the need for secrecy G09C) ## Abstract In various embodiments, methods and systems for implementing distributed data object management are provided. The distributed data object management system includes a distributed storage system having a local metadata-consensus information store in and one or more remote metadata-consensus information stores. A metadata-consensus information store is configured to store metadata-consensus information. The metadata-consensus information corresponds to erasure coded fragments of a data object and instruct on how to manage the erasure coded fragments. The distributed storage system further includes a local data store and one or more remote data stores for the erasure coded fragments. The distributed data object management system includes a distributed data object manager for operations including, interface operations, configuration operations, write operations, read operations, delete operations, garbage collection operations and failure recovery operations. The distributed data object management system is operates based on metadata paths and data paths, operating in parallel, for write operations and read operations. ## Claims ### Claim 1 A system for implementing distributed data object management, the system comprising: a distributed storage system comprising: a local metadata-consensus information store and one or more remote metadata-consensus information stores, wherein a metadata-consensus information store is a table store for metadata-consensus information, the metadata-consensus information corresponds to erasure coded fragments of a data object and instructs on how to manage the erasure coded fragments; a local data store and one or more remote data stores, wherein a data store is an object store that stores the erasure coded fragments of the data object, the local data store and the one or more remote data stores store the erasure coded fragments of the data object that correspond to the metadata-consensus information in the local metadata-consensus information store and the one or more remote metadata-consensus information stores, wherein corresponding metadata writes and data writes for a write operation are performed in parallel using a metadata write path and a data write path, respectively, when writing to the local metadata-consensus information store and the one or more remote metadata-consensus information stores and the local data store and the one or more remote data stores; and wherein corresponding metadata reads and data reads for a read operation are performed in parallel using a metadata read path and a data read path, respectively, when reading from the local metadata-consensus information store and the one or more remote metadata-consensus information stores and the local data store and the one or more remote data stores. ### Claim 2 (depends on 1) The system of claim 1, wherein the metadata-consensus information comprises one or more of the following: a known committed version, wherein the known committed version operates as a hint element in write operations and read operations; a pointer to the corresponding erasure coded fragments of the data object; one or more triplets of version column instances, a triplet of version columns comprising: a highest ballot number seen; a highest accepted ballot number; and a highest accepted value. ### Claim 3 (depends on 1) The system of claim 1, further comprising a distributed data object manager configured to execute interface operations, the interface operations comprising: providing a customer access to a storage account associated with data objects of the customer; receiving a selection of a set of data centers where erasure coded fragments of the data objects are to be allocated; and receiving a selection of an erasure coding scheme, wherein the erasure coding scheme is used to generate erasure coded fragments of the data objects, wherein, based on the erasure coding scheme, the erasure coded fragments of the data objects comprise a defined number of data fragments and a defined number of parity fragments. ### Claim 4 (depends on 1) The system of claim 1, further comprising a distributed data object manager configured to execute configuration operations, the configuration operations comprising: accessing a data availability profile of a customer, wherein the data availability profile identifies availability parameters selected for managing availability of data objects of the customer; based on the data availability profile, determining a number of data centers for storing erasure coded fragments and metadata-consensus information of the data objects of the customer, wherein a mapping configuration for the erasure coded fragments and metadata-consensus information indicates a mapping for storing the erasure coded fragments and metadata-consensus information in the data centers; accessing an indication of a configuration change trigger to change the mapping configuration; and based on accessing the indication of the configuration change trigger, changing the mapping configuration to a new mapping configuration, the new mapping configuration is generated based at least in part on a grace period where the mapping configuration previously being used is invalidated. ### Claim 5 (depends on 1) The system of claim 1, further comprising a distributed data object manager configured to execute failure recovery operations comprising: receiving an indication of a transient data center failure for a given data center, wherein table stores and object stores of the given data center are temporarily not accessible; initiating processing of write operations and read operations based on degraded write operations and degraded read operations, respectively, wherein a degraded write operation comprises at least two cross-data-center roundtrips; and wherein a degraded read operation comprises reading at least a parity fragment of the erasure coded fragments of the data object; receiving an indication that the given data center has recovered from the transient data center failure; and triggering an update of table stores and object stores of the given data center based on: reading the erasure coded fragments of the data object from other object stores not at the given data center; recalculating an erasure coded fragment of the data object that belongs to the object store at the given data center; and writing the erasure coded fragment of the data object to the object store. ### Claim 6 (depends on 1) The system of claim 1, further comprising a distributed data object manager configured to execute failure recovery operations comprising: receiving an indication of a permanent data center failure for a given data center, wherein table stores and object stores for the given data center are permanently not accessible; triggering a data center configuration change to replace the given data center; and causing regeneration of lost erasure coded fragments of the data object. ### Claim 7 (depends on 1) The system of claim 1, further comprising a distributed data object manager configured to execute delete operations comprising one of the following: trimming earlier version instances of the data object, wherein a version instance comprises stored metadata-consensus information and corresponding erasure coded fragments of a version of the data object, wherein trimming earlier version instances is based on: determining that a version instance count limit has been met; writing a new version instance of the data object; and automatically trimming an earliest version instance of the data object; and deleting a specific version instance of a data object, wherein deleting the specific version instance is based on: executing a write operation to create a delete record for the specific version instance in the local metadata-consensus information store and the one or more remote metadata-consensus information stores, wherein the delete record supports deleting metadata-consensus information for the specific version instance, and wherein the delete record operates as an indicator for the erasure coded fragments of the specific version instance to be permanently deleted using a garbage collection operation; and deleting an object instance of the data object, wherein an object instance comprises all stored metadata-consensus information and corresponding erasure coded fragments of a version of the data object, wherein deleting the object instance is based on: executing a tombstone marker write operation, wherein the tombstone marker write operation commits a tombstone marker version as a newest version operating as an indicator to delete the object instance; deleting the erasure coded fragments corresponding to the object instance; and deleting table rows of the metadata-consensus information corresponding to the object instance. ### Claim 8 (depends on 1) The system of claim 1, further comprising a distributed data object manager configured to execute garbage collection operations comprising truncating corresponding triplet version column instances of erasure coded fragments and deleting their corresponding erasure coded fragments. ### Claim 9 (depends on 1) The system of claim 1, wherein executing the write operation and the read operation is based on a consensus protocol, wherein the consensus protocol is comprises a fast-path protocol and a slow path protocol for using the metadata write path, data write path, metadata read path and data read path, such that, writes and reads in common-cases are performed using the fast-path protocol and writes and reads in contention-cases are performed using the slow-path protocol. ### Claim 10 A computer-implemented method for distributed data object management, the method comprising: executing a delete operation in a distributed storage system, the delete operation comprises one of the following: trimming earlier version instances of a data object, wherein a version instance comprises stored metadata-consensus information and corresponding erasure coded fragments of a version of the data object; deleting a specific version instance of the data object; or deleting an object instance of the data object, wherein an object instance comprises all stored metadata-consensus information and corresponding erasure coded fragments of the data object of a version of the data object; and wherein the distributed storage system comprises a local metadata-consensus information store and one or more remote metadata-consensus information stores, wherein a metadata-consensus information store stores metadata-consensus information, the metadata-consensus information corresponds to erasure coded fragments of the data object and instructs on how to manage the erasure coded fragments; and a local data store and one or more remote data stores, wherein a data store stores the erasure coded fragments of the data object, the local data store and the one or more remote data stores store the erasure coded fragments of the data object that correspond to the metadata-consensus information in the local metadata-consensus information store and the one or more remote metadata-consensus information stores. ### Claim 11 (depends on 10) The method of claim 10, wherein trimming earlier version instances is based on: determining that a version instance count limit has been met; writing a new version instance of the data object; and automatically trimming an earliest version instance of the data object. ### Claim 12 (depends on 10) The method of claim 10, wherein deleting the specific version instance is based on: executing a write operation to create a delete record for the specific version instance in the local metadata-consensus information store and the one or more remote metadata-consensus information stores, wherein the delete record supports deleting metadata-consensus information for the specific version instance, and wherein the delete record operates as an indicator for the erasure coded fragments of the specific version instance to be permanently deleted using a garbage collection operation. ### Claim 13 (depends on 10) The method of claim 10, wherein deleting the object instance is based on: executing a tombstone marker write operation, wherein the tombstone marker write operation commits a tombstone marker version as a newest version operating as an indicator to delete the object instance; deleting the erasure coded fragments corresponding to the object instance; and deleting table rows of the metadata-consensus information corresponding to the object instance. ### Claim 14 (depends on 13) The method of claim 13, wherein deleting the table rows of the metadata-consensus information is based on a two-phase commit deletion operation comprising: marking the table rows of the metadata-consensus information to designate the table rows for deletion, such that, write and read operations for the object instance are temporarily disabled; and removing the table rows of the metadata-consensus information. ### Claim 15 (depends on 10) The method of claim 10, wherein corresponding metadata writes and data writes for a write operation are performed in parallel using a metadata write path and a data write path, respectively, when writing to the local metadata-consensus information store and the one or more remote metadata-consensus information stores and the local data store and the one or more remote data stores; and wherein corresponding metadata reads and data reads for a read operation are performed in parallel using a metadata read path and a data read path, respectively, when reading from the local metadata-consensus information store and the one or more remote metadata-consensus information stores and the local data store and the one or more remote data stores. ### Claim 16 A computer-implemented method for distributed data object management, the method comprising: executing a garbage collection operation in a distributed storage system, the garbage collection operation comprises deleting triplet version column instances and deleting corresponding erasure coded fragments based on: deleting metadata-consensus information in a local metadata-consensus information store and one or more remote metadata-consensus information stores, wherein a metadata-consensus information store stores metadata-consensus information, the metadata-consensus information corresponds to erasure coded fragments of a data object and instructs on how to manage the erasure coded fragments; and deleting erasure coded fragments of the data object in a local data store and one or more remote data stores, wherein a data store stores the erasure coded fragments of the data object, the local data store and the one or more remote data stores store the erasure coded fragments of the data object that correspond to the metadata-consensus information in the local metadata-consensus information store and the one or more remote metadata-consensus information stores. ### Claim 17 (depends on 16) The method of claim 16, wherein deleting the metadata-consensus information and deleting the erasure coded fragments is based on: identifying, from corresponding table stores of the metadata-consensus information in the local metadata-consensus information store and the one or more remote metadata-consensus information stores, a metadata table row corresponding to a version instance of the metadata-consensus information to be deleted; sending deletion requests to the local data store and the one or more remote data stores, to delete the corresponding erasure coded fragments; and removing corresponding triplet version column instances of the version instance from local metadata-consensus information store and the one or more remote metadata-consensus information stores. ### Claim 18 (depends on 16) The system of claim 16, wherein a triplet version column instance for a selected version instance comprises: a highest ballot number seen; a highest accepted ballot number; and a highest accepted value. ### Claim 19 (depends on 16) The system of claim 16, wherein garbage collection is performed for a specific version instance or an object instance comprising a plurality of version instances. ### Claim 20 (depends on 16) The method of claim 16, wherein corresponding metadata writes and data writes for a write operation are performed in parallel using a metadata write path and a data write path, respectively, when writing to the local metadata-consensus information store and the one or more remote metadata-consensus information stores and the local data store and the one or more remote data stores; and wherein corresponding metadata reads and data reads for a read operation are performed in parallel using a metadata read path and a data read path, respectively, when reading from the local metadata-consensus information store and the one or more remote metadata-consensus information stores and the local data store and the one or more remote data stores.