Network system to distribute chunks across multiple physical nodes with disk support for object storage

A non-transitory computer-readable medium that stores a computer program for storing a file in a storage system that includes a plurality of memory-storage hosts, the computer program including instructions that, when executed by at least one computer, cause the at least one computer to perform operations comprising:

• specifying a unique chunk identifier for a memory chunk included in the file;
• specifying a hash mapping to identify a plurality of storage locations for the chunk identifier, each memory-storage host including a corresponding disk-storage unit, and each storage location corresponding to a portion of a disk-storage unit included in a memory-storage host, the hash mapping including host-identification outputs and host-storage outputs, the host-identification outputs identifying a plurality of the memory-storage hosts for storing the memory chunk, and the host-storage outputs identifying one or more storage locations at each identified memory-storage host;
• providing metadata corresponding to a first portion of the hash mapping to a memory-management unit for the storage system, the metadata corresponding to the first portion of the hash mapping being used to access the first portion of the hash mapping, and the memory management unit being configured to implement the first portion of the hash mapping by determining the host-identification outputs from the hash mapping;
• providing metadata corresponding to a second portion of the hash mapping to a disk-storage interface of a corresponding disk-storage unit for each memory-storage host of the plurality of memory storage hosts, the metadata corresponding to the second portion of the hash mapping being used to access the second portion of the hash mapping, and the disk-storage interface being configured to implement the second portion of the hash mapping by determining the host-storage outputs that identify storage locations at that memory-storage host;
• maintaining, at each memory-storage host of the plurality of memory-storage hosts, a shared memory with the memory-management unit, the shared memory including one or more buffers for accessing storage locations of that memory-storage host by the memory-management unit;
• and
• storing the memory chunk at the storage locations identified by the hash mapping by using the memory-management unit to determine the host-identification outputs from the hash mapping and to provide the chunk identifier to the shared memory of each identified memory-storage host corresponding to the host-identification outputs, and by using each identified memory-storage host to determine its corresponding host-storage outputs from the hash mapping and to store the memory chunk at the identified one or more storage locations corresponding to its host-storage outputs via its shared memory with the memory-management unit.

A system storing a file in a storage system that includes a plurality of memory-storage hosts, the system including at least one hardware processor configured to perform operations for hardware-implemented modules including:

• a chunk-identifier module configured to specify a unique chunk identifier for a memory chunk included in the file;
• a hash-mapping module configured to specify a hash mapping to identify a plurality of storage locations for the chunk identifier, each memory-storage host including a corresponding disk-storage unit, each storage location corresponding to a portion of a disk-storage unit included in a memory-storage host, the hash mapping including host-identification outputs and host-storage outputs, the host-identification outputs identifying a plurality of the memory-storage hosts for storing the memory chunk, and the host-storage outputs identifying one or more storage locations at each identified memory-storage host;
• a first implementation module configured to provide metadata corresponding to a first portion of the hash mapping to a memory-management unit for the storage system, the metadata corresponding to the first portion of the hash mapping being used to access the first portion of the hash mapping, and the memory management unit being configured to implement the first portion of the hash mapping by determining the host-identification outputs from the hash mapping;
• a second implementation module configured to provide metadata corresponding to a second portion of the hash mapping to a disk-storage interface of a corresponding disk-storage unit for each memory-storage host of the plurality of memory storage hosts, the metadata corresponding to the second portion of the hash mapping being used to access the second portion of the hash mapping, and the disk-storage interface being configured to implement the second portion of the hash mapping by determining the host-storage outputs that identify storage locations at that memory-storage host;
• a shared-memory module configured to maintain, at each memory-storage host of the plurality of memory-storage hosts, a shared memory with the memory-management unit, the shared memory including one or more buffers for accessing storage locations of that memory storage host by the memory-management unit;
• and
• a storage module configured to store the memory chunk at the storage locations identified by the hash mapping by using the memory-management unit to determine the host-identification outputs from the hash mapping and to provide the chunk identifier to the shared memory of each identified memory-storage host corresponding to the host-identification outputs, and by using each identified memory-storage host to determine its corresponding host-storage outputs from the hash mapping and to store the memory chunk at the identified one or more storage locations corresponding to its host-storage outputs via its shared memory with the memory-management unit.