Method for reading and writing data and distributed storage system

A data read and write method, applied to a master server in a distributed storage system, the distributed storage system comprising:

• a first shard server, a second shard server, the master server, and a distributed file system, wherein the first and second shard servers are hardware-based servers,
• wherein data to be read and written includes pieces of shard data, and each piece of shard data is assigned a data index comprising a storage location of the each piece of shard data in the distributed file system,
• wherein each piece of shard data has a data identifier associated with it, and each piece of shard data has the data index assigned to it, and a correspondence between the data identifier and the data index is recorded in a hash table,
• wherein hash values of data identifiers of shard data are partitioned into different shards based on different hash value ranges, and each shard is of a hash value range different from each other,
• wherein the first shard server includes a memory with the hash table preloaded in the memory, and the first shard server further including a plurality of primary shard modules that are software modules, wherein each primary shard module in the plurality of primary shard modules is assigned a shard different to each other and is assigned a hash value range and is responsible to process read and write requests to a shard in the respective primary shard module with the hash value range that the each primary shard module is assigned,
• wherein the second shard server includes a plurality of secondary shard modules that are software modules, wherein each secondary shard module in the plurality of secondary shard modules is a backup of a corresponding primary shard module on the first server, the each secondary shard module is responsible to process read and write requests to a shard in the respective secondary shard module with a hash value range that is the same as a hash value range that the corresponding primary shard module in the first server is assigned, and each secondary shard module is assigned a shard different to each other,
• wherein the method comprises:
• receiving a query request sent from a client, wherein the query request comprises a data identifier corresponding to to-be-processed shard data;
• and
• determining a shard server used for processing the to-be-processed shard data based on a hash value range that a hash value corresponding to the data identifier falls in, so that the client is capable of sending a processing request on the shard data to the determined shard server for processing the to-be-processed shard data, comprising:
• determining the shard server where one of the plurality of the primary shard modules or one of the plurality of the secondary shard modules is located as the shard server used for processing the to-be-processed shard data, wherein said one of the plurality of the primary shard modules and said one of the plurality of the secondary shard modules are determined based on a hash value range that a hash value corresponding to the data identifier falls in, and said primary shard module and said secondary shard module are located on different shard servers and are preset to correspond to an identical hash value range, wherein each one of the plurality of the primary shard modules and the plurality of the secondary shard modules reads data from or writes data to the distributed file system,
• wherein the method further comprises:
• in response to the processing request being a write request, the one of the plurality of the primary shard modules or the one of the plurality of secondary shard modules writes the shard data to the distributed file system.

A method for reading data from and writing data to a distributed file system in a distributed storage system, the distributed storage system comprising a first shard server, a second shard server, a master server, and the distributed file system, wherein the first and second shard servers are hardware-based servers,

• wherein data to be read and written includes pieces of shard data, and each piece of shard data is assigned a data index comprising a storage location of the each piece of shard data in the distributed file system,
• wherein each piece of shard data has a data identifier associated with it, and each piece of shard data has the data index assigned to it, and a correspondence between the data identifier and the data index is recorded in a hash table,
• wherein hash values of data identifiers of shard data are partitioned into different shards based on different hash value ranges, and each shard is of a hash value range different from each other,
• wherein the first shard server includes a memory with the hash table preloaded in the memory, and the first shard server further including a plurality of primary shard modules that are software modules, wherein each primary shard module in the plurality of primary shard modules is assigned a shard different to each other and is assigned a hash value range and is responsible to process read and write requests to a shard in the respective primary shard module with the hash value range that the each primary shard module is assigned,
• wherein the second shard server includes a plurality of secondary shard modules that are software modules, wherein each secondary shard module in the plurality of secondary shard modules is a backup of a corresponding primary shard module on the first server, the each secondary shard module is responsible to process read and write requests to a shard in the respective secondary shard module with a hash value range that is the same as a hash value range that the corresponding primary shard module in the first server is assigned, and each secondary shard module is assigned a shard different to each other,
• wherein the method comprises:
• receiving a processing instruction on shard data, and sending a query request to the master server, the query request comprising a data identifier of the shard data;
• sending a processing request on the shard data to the first shard server or the second shard server, the first shard server and the second shard server being determined by the master server based on the data identifier of the shard data;
• and
• receiving a processing result returned from the first shard server after the first shard server processes the processing request, wherein the processing request is processed by using one of the plurality of the primary shard modules located on the first shard server or one of the plurality of the secondary shard modules located on the second shard server, said one of the plurality of the primary shard modules and said one of the plurality of the primary shard modules are determined based on a hash value range that a hash value corresponding to the data identifier of the piece of shard data falls in, and said primary shard module and said secondary shard module are preset to correspond to an identical hash value range,
• wherein the method further comprises:
• in response to the processing request being a write request, the one of the plurality of the primary shard modules or the one of the plurality of secondary shard modules writes the shard data to the distributed file system.

A distributed storage system, comprising a first shard server, a second shard server, and a distributed file system, wherein the first and second shard servers are hardware-based servers,

• wherein data to be read and written includes pieces of shard data, and each piece of shard data is assigned a data index comprising a storage location of the each piece of shard data in the distributed file system,
• wherein each piece of shard data has a data identifier associated with it, and each piece of shard data has the data index assigned to it, and a correspondence between the data identifier and the data index is recorded in a hash table,
• wherein hash values of data identifiers of shard data are partitioned into different shards based on different hash value ranges, and each shard is of a hash value range different from each other,
• wherein the first shard server includes a memory with the hash table preloaded in the memory, and the first shard server further including a plurality of primary shard modules that are software modules, wherein each primary shard module in the plurality of primary shard modules is assigned a shard different to each other and is assigned a hash value range and is responsible to process read and write requests to a shard in the respective primary shard module with the hash value range that the each primary shard module is assigned,
• wherein the second shard server includes a plurality of secondary shard modules that are software modules, wherein each secondary shard module in the plurality of secondary shard modules is a backup of a corresponding primary shard module on the first server, the each secondary shard module is responsible to process read and write requests to a shard in the respective secondary shard module with a hash value range that is the same as a hash value range that the corresponding primary shard module in the first server is assigned, and each secondary shard module is assigned a shard different to each other,
• the distributed storage system is configured to perform operations comprising:
• receiving, by a client, a processing instruction on a piece of shard data, and sending a query request to a master server, the query request comprising a data identifier of the piece of shard data;
• sending a processing request on the piece of shard data to the first shard server, the first shard server being determined by the master server based on the data identifier of the piece of shard data;
• and receiving a processing result returned from the first shard server after the first shard server processes the processing request;
• receiving, by the master server, the query request sent from the client, wherein the query request comprises the data identifier corresponding to the piece of shard data;
• and
• determining the first shard server used for processing the piece of shard data based on the hash value range that the hash value corresponding to the data identifier falls in, so that the client is capable of sending the processing request on the piece of shard data to the first shard server for processing the piece of shard data;
• and
• receiving, from the client, by the first shard server, the processing request on the piece of shard data, the processing request comprising the data identifier of the piece of shard data, and the processing request being a write request or a read request;
• processing the processing request based on the hash table pre-loaded in a memory of the first shard server to obtain a processing result;
• and sending the processing result to the client,
• the processing request is processed by using one of the plurality of the primary shard modules located on the first shard server or one of the plurality of the secondary shard modules located on the second shard server, said one of the plurality of the primary shard modules and said one of the plurality of the primary shard modules are determined based on a hash value range that a hash value corresponding to the data identifier of the piece of shard data falls in, and said primary shard module and said secondary shard module are preset to correspond to an identical hash value range,
• wherein the operations further comprise:
• in response to the processing request being a write request, the one of the plurality of the primary shard modules or the one of the plurality of secondary shard modules writes the piece of shard data to the distributed file system.