# US 12,001,380 B2 · Single-sided distributed storage system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,380 B2
- Document ID: US12001380B2
- Status: Granted
- Kind: B2
- Application number: 18/298,164
- Filed: 2023-04-10
- Published: 2023-08-03
- Granted: 2024-06-04
- Earliest claimed: 2012-06-08
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Kyle Nesbit; Andrew Everett Phelps
- Web page: https://patentgrep.com/patent/US12001380B2

## CPC

- G06F 15/17331: {Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]}
- G06F 3/0611: {in relation to response time}
- G06F 3/0619: {in relation to data integrity, e.g. data losses, bit errors}
- G06F 3/0643: {Management of files}
- G06F 3/065: {Replication mechanisms}
- G06F 3/0659: {Command handling arrangements, e.g. command buffers, queues, command scheduling}
- G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A distributed storage system including memory hosts and at least one curator in communication with the memory hosts. Each memory host has memory, and the curator manages striping of data across the memory hosts. In response to a memory access request by a client in communication with the memory hosts and the curator, the curator provides the client a file descriptor mapping data stripes and data stripe replications of a file on the memory hosts for remote direct memory access of the file on the memory hosts.

## Claims

### Claim 1

A computer-implemented method when executed by data processing hardware of a distributed storage system causes the data processing hardware to perform operations comprising:
dividing a file into a plurality of data stripes;
for each respective data stripe of the plurality of data stripes, allocating storage of the respective data stripe to a respective memory host of a plurality of memory hosts of the distributed storage system;
receiving, from a client, a memory access request requesting access to the file;
in response to the memory access request, retrieving a file descriptor mapping each respective data stripe to the respective memory host of the plurality of memory hosts for access of the file by the client;
receiving an external notification indicating that an attempt to access a particular data stripe of the plurality of data stripes stored at a particular memory host of the plurality of memory hosts has failed; and
in response to receiving the external notification, reconstructing the particular data stripe.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise, in response to the memory access request, returning a key allowing access to data on the plurality of memory hosts.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the file descriptor comprises an array of stripe protocol buffers, each stripe protocol buffer describing a particular replica data stripe within a particular respective data stripe.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise determining a state of each respective data stripe of the plurality of data stripes, the state indicating whether the respective data stripes is available for allocation to a respective destination storage location.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise receiving an indication that the client was able to access the file.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the memory access request comprises a RDMA read network operation.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise:
receiving, from the client, a second memory access request;
in response to the second memory access request, determining that at least one of the plurality of data stripes is migrating to a respective destination storage location of the plurality of memory hosts; and
denying the second memory access request.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein each respective memory host of the plurality of memory hosts comprises a network interface controller in communication with a memory of the respective memory host, the network interface controller servicing remote direct memory access requests.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the file descriptor comprises one or more of:
a file state attribute indicating a state of the file;
a data chunks attribute indicating a number of replica data stripes per respective data stripe;
a stripe length attribute indicating a number of bytes per respective data stripe; or
a sub-stripe length attribute indicating a number of bytes per sub-stripe in the file descriptor.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise:
accessing a file map mapping files to a plurality of file descriptors; and
in response to the memory access request, returning the file descriptor from the plurality of file descriptors.

### Claim 11

A system comprising:
data processing hardware of a distributed storage system; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
dividing a file into a plurality of data stripes;
for each respective data stripe of the plurality of data stripes, allocating storage of the respective data stripe to a respective memory host of a plurality of memory hosts of the distributed storage system;
receiving, from a client, a memory access request requesting access to the file;
in response to the memory access request, retrieving a file descriptor mapping each respective data stripe to the respective memory host of the plurality of memory hosts for access of the file by the client;
receiving an external notification indicating that an attempt to access a particular data stripe of the plurality of data stripes stored at a particular memory host of the plurality of memory hosts has failed; and
in response to receiving the external notification, reconstructing the particular data stripe.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise, in response to the memory access request, returning a key allowing access to data on the plurality of memory hosts.

### Claim 13 (depends on 11)

The system of claim 11, wherein the file descriptor comprises an array of stripe protocol buffers, each stripe protocol buffer describing a particular replica data stripe within a particular respective data stripe.

### Claim 14 (depends on 11)

The system of claim 11, wherein the operations further comprise determining a state of each respective data stripe of the plurality of data stripes, the state indicating whether the respective data stripes is available for allocation to a respective destination storage location.

### Claim 15 (depends on 11)

The system of claim 11, wherein the operations further comprise receiving an indication that the client was able to access the file.

### Claim 16 (depends on 11)

The system of claim 11, wherein the memory access request comprises a RDMA read network operation.

### Claim 17 (depends on 11)

The system of claim 11, wherein the operations further comprise:
receiving, from the client, a second memory access request;
in response to the second memory access request, determining that at least one of the plurality data stripes is migrating to a respective destination storage location of the plurality of memory hosts; and
denying the second memory access request.

### Claim 18 (depends on 11)

The system of claim 11, wherein each respective memory host of the plurality of memory hosts comprises a network interface controller in communication with a memory of the respective memory host, the network interface controller servicing remote direct memory access requests.

### Claim 19 (depends on 11)

The system of claim 11, wherein the file descriptor comprises one or more of:
a file state attribute indicating a state of the file;
a data chunks attribute indicating a number of replica data stripes per respective data stripe;
a stripe length attribute indicating a number of bytes per respective data stripe; or
a sub-stripe length attribute indicating a number of bytes per sub-stripe in the file descriptor.

### Claim 20 (depends on 11)

The system of claim 11, wherein the operations further comprise:
accessing a file map mapping files to a plurality of file descriptors; and
in response to the memory access request, returning the file descriptor from the plurality of file descriptors.