Systems and methods of handling node failure in a distributed data storage using multi-layer consistent hashing

A distributed data storage system using multiple-layers consistent hashing, comprising:

• a plurality of storage nodes providing data storage and redundancy protection, wherein the storage nodes form a storage hierarchical tree, and direct child nodes of a parent node share a same hash space;
• a plurality of management nodes maintaining properties of the storage nodes and mapping information from virtual groups to storage nodes, wherein each of the virtual groups corresponds to a partition of hash space, that is, a hash subspace;
• wherein the mapping information from virtual groups to storage nodes is based on hashing, and comprises:
• mapping information from virtual groups to qualified storage nodes list;
• and
• mapping information from virtual groups and a failed storage node to a replacement node of the failed storage node;
• wherein the qualified storage nodes list comprises a primary node and one or more secondary nodes;
• a plurality of monitor nodes maintaining states of the storage nodes and handling changes of states of the storage nodes including joining, decommissioning and failure, the monitor nodes retrieve state of the storage nodes in the qualified storage nodes list, and the primary node and a secondary node report a failure of each other to the monitor nodes;
• and
• one or more clients providing entries for applications or users to access the storage system, wherein the client is used for initiating an access request of an object and searching the primary node corresponding to the object with a two-phase mapping, wherein a first mapping is mapping an object to a virtual group used as an object container, and a second mapping is mapping from a virtual group to a qualified nodes list, then realizes process of object access comprising:
• an object writing step:
• the client uploading a new object to the storage system;
• an object reading step:
• the client downloading an object from the storage system;
• an object updating step:
• the client modifying an existed object in the storage system;
• and
• an object deleting step:
• the client deleting an existed object in the storage system.