# US 10,484,472 B2 · Methods and systems for efficiently moving data between nodes in a cluster

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,484,472 B2
- Document ID: US10484472B2
- Status: Granted
- Kind: B2
- Application number: 14/840,512
- Filed: 2015-08-31
- Published: 2017-02-02
- Granted: 2019-11-19
- Earliest claimed: 2015-07-31
- Assignees: NetApp, Inc.
- Applicants: Netapp, Inc.
- Inventors: Balajee Nagasubramaniam; Subin Govind; Vijay Singh; Jeff Prem
- Web page: https://patentgrep.com/patent/US10484472B2

## CPC

- H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
- G06F 15/167: using a common memory, e.g. mailbox
- H04L 43/16: Threshold monitoring
- H04L 69/16: Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L 69/161: {Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields}
- H04L 69/18: Multiprotocol handlers, e.g. single devices capable of handling multiple protocols
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Exemplary embodiments provide methods, mediums, and systems for efficiently moving data between cluster nodes. Upon receiving a request to read or write data at a first cluster node that is in communication with a client, the first node effects the transfer to or from a second cluster node. The transfer is carried out using a combination of remote data memory access (“RDMA”), or a similar technique that bypasses a part of the network stack, and transport control protocol (“TCP”), or a similar technique that does not bypass a part of the network stack. The data is transferred using RDMA, while certain control messages are sent using TCP. By combining RDMA content transfers and TCP control messages, data transfers can be carried out faster, more efficiently, and with less processing overhead. Other embodiments are described and claimed.

## Claims

### Claim 1

A method comprising:
receiving, via a transport control protocol (TCP) by a first node, a read request from a client device to read data, comprising metadata and content, that is stored on a remote volume associated with a second node content;
allocating, by the first node, a buffer within memory of the first node for receiving the content using remote direct memory access (RDMA) based upon a size of the content exceeding a size threshold and resource consumption for allocating the buffer being below a threshold;
transmitting an address of the buffer to the second node via the TCP to trigger the second node to perform an RDMA write operation to write the content into the buffer using the address;
receiving, via the TCP by the first node from the second node, the metadata comprising instructions for reconstructing the data using the content within the buffer, wherein a response header comprises an indication of whether the RDMA write operation was successful; and
reconstructing and transmitting the data to the client device using the metadata and the content based upon the instructions.

### Claim 2 (depends on 1)

The method of claim 1, comprising:
deallocating the buffer from the memory based upon transmitting the data to the client device.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
extract the content from the buffer based upon the flag indicating that the RDMA write operation by the second node wrote the content into the buffer within the memory of the first node.

### Claim 4 (depends on 1)

The method of claim 1, wherein the reconstructing comprises:
combining the metadata received via the TCP and the content received through the buffer via the RDMA to construct the data.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
receiving the content via TCP as opposed to the RDMA when the size of the content is less than the size threshold.

### Claim 6 (depends on 1)

The method of claim 1, wherein the size threshold is between about 16 kilobytes and about 32 kilobytes.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
reverting to data transmission via the TCP when communication via the RDMA is impossible.

### Claim 8 (depends on 1)

The method of claim 1, wherein the RDMA write operation is performed by the second node to facilitate execution of the read request by the first node.

### Claim 9

A non-transitory computer readable medium storing instructions that, when executed, cause circuitry of a computing device to:
receive, via a transport control protocol (TCP) by a first node, a read request from a client device to read data, comprising metadata and content, that is stored on a remote volume associated with a second node;
allocate, by the first node, a buffer within memory of the first node for receiving the content using remote direct memory access (RDMA) based upon a size of the content exceeding a size threshold and resource consumption for allocating the buffer being below a threshold;
transmit an address of the buffer to the second node using the TCP to trigger the second node to perform an RDMA write operation to write the content into the buffer using the address;
receive, via the TCP by the first node from the second node, the metadata comprising instructions for reconstructing the data using the content within the buffer, wherein a response header comprises an indication of whether the RDMA write operation was successful; and
reconstruct and transmit the data to the client device using the metadata and the content based upon the instructions.

### Claim 10 (depends on 9)

The medium of claim 9, wherein the instructions cause the computing device:
deallocate the buffer from the memory based upon transmitting the data to the client device.

### Claim 11 (depends on 9)

The medium of claim 9, wherein the instructions cause the computing device to:
extract the content from the buffer based upon the flag indicating that the RDMA write operation by the second node wrote the content into the buffer within the memory of the first node.

### Claim 12 (depends on 9)

The medium of claim 9, wherein the instructions cause the computing device:
combine the metadata received via the TCP and the content received through the buffer via the RDMA to construct the data.

### Claim 13 (depends on 9)

The medium of claim 9, wherein the instructions cause the computing device:
receiving the content via TCP as opposed to the RDMA when the size of the content is less than the size threshold.

### Claim 14 (depends on 9)

The medium of claim 9, wherein the instructions cause the computing device:
revert to data transmission via the TCP when communication via the RDMA is impossible.

### Claim 15 (depends on 9)

The medium of claim 9, wherein the RDMA write operation is performed by the second node to facilitate execution of the read request by the first node.

### Claim 16

A computing device, comprising:
a memory comprising machine executable code; and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to:
receive, via a transport control protocol (TCP) by a first node, a read request from a client device to read data, comprising metadata and content, that is stored on a remote volume associated with a second node;
allocate, by the first node, a buffer within memory of the first node for receiving the content using remote direct memory access (RDMA) based upon a size of the content exceeding a size threshold and resource consumption for allocating the buffer being below a threshold;
transmit an address of the buffer to the second node using the TCP to trigger the second node to perform an RDMA write operation to write the content into the buffer using the address;
receive, via the TCP by the first node from the second node, the metadata comprising instructions for reconstructing the data using the content within the buffer, wherein a response header comprises an indication of whether the RDMA write operation was successful; and
reconstruct and transmit the data to the client device using the metadata and the content based upon the instructions.

### Claim 17 (depends on 16)

The computing device of claim 16, wherein the buffer is deallocated from the memory based upon transmitting the data to the client device.

### Claim 18 (depends on 16)

The computing device of claim 16, wherein the content is transmitted via the TCP when the size of the content is less than the size threshold.

### Claim 19 (depends on 18)

The computing device of claim 18, wherein the size threshold is between about 16 kilobytes and about 32 kilobytes.

### Claim 20 (depends on 16)

The computing device of claim 16, wherein the machine executable code causes the processor to revert to data transmission via the TCP when communication via the RDMA is impossible.