Multi-level lookup architecture to facilitate failure recovery

A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for operating a data storage system, the method comprising:

• receiving a request to retrieve a data block from the data storage system, wherein the request includes a hash that functions as a global identifier for the data block;
• using the hash to look up a bucket and an associated cell that contains the data block, wherein the bucket aggregates a large number of data blocks, wherein the bucket is located in the associated cell that comprises a set of object storage devices (OSDs), and wherein the lookup is performed in a hash database (HDB) for the data storage system;
• within the cell, using the bucket to look up an OSD that contains the bucket, wherein the lookup is performed in a local bucket database (BDB) for the cell;
• within the OSD, using the bucket and the hash to determine an offset and a length for the data block in a write-ahead log that stores data blocks for the bucket;
• and
• returning the data block from the determined offset in the write-ahead log.

A data storage system, comprising:

• one or more zones, wherein each zone is associated with a data center;
• wherein each zone is divided into a plurality of cells;
• wherein each cell comprises a plurality of object storage devices (OSDs) for storing blocks of data, and a master for managing the OSDs;
• and
• wherein during operation, the data storage system is configured to,
• receive a request to retrieve a data block from the data storage system, wherein the request includes a hash that functions as a global identifier for the data block,
• use the hash to look up a bucket and an associated cell that contains the data block, wherein the bucket aggregates a large number of data blocks and is located in the associated cell, and wherein the lookup is performed in a hash database (HDB) for the data storage system;
• within the cell, use the bucket to look up an OSD that contains the bucket, wherein the lookup is performed in a local bucket database (BDB) for the cell;
• within the OSD, use the bucket and the hash to determine an offset and a length for the data block in a write-ahead log that stores data blocks for the bucket;
• and
• return the data block from the determined offset in the write-ahead log.