# US 2021/0081352 A1 · Internet small computer interface systems extension for remote direct memory access (rdma) for distributed hyper-converged storage systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081352 A1
- Document ID: US20210081352A1
- Status: Published application
- Kind: A1
- Application number: 16/656,499
- Filed: 2019-10-17
- Published: 2021-03-18
- Earliest claimed: 2019-09-17
- Assignees: VMware, Inc.
- Applicants: VMware, Inc.
- Inventors: Yang YANG; Haitao ZHOU; Zhaohui GUO; Jian ZHAO; Zhou HUANG; Jin FENG
- Web page: https://patentgrep.com/patent/US20210081352A1

## CPC

- G06F 15/17331: {Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]}
- G06F 13/1668: {Details of memory controller}
- G06F 13/4221: {being an input/output bus, e.g. ISA bus, EISA bus, PCI bus, SCSI bus}
- H04L 49/901: using storage descriptor, e.g. read or write pointers
- H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
- H04L 69/22: Parsing or analysis of headers
- H04L 69/321: Interlayer communication protocols or service data unit [SDU] definitions; Interfaces between layers
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Certain Embodiments described herein relate to configuring the network-storage stack of two devices (e.g., physical or virtual) communicating together (e.g., an initiator and a target, as defined below) with Internet Small Computer Systems Interface (iSCSI) extension for remote direct memory access (RDMA) iSER, which is a protocol designed to utilize RDMA to accelerate iSCSI data transfer. The iSER protocol is implemented as an iSER datamover layer that acts as an interface between an iSCSI layer and an RDMA layer of the network-storage stacks of the two devices. Using iSER in conjunction with RDMA allows for bypassing the existing traditional network protocol layers (e.g., TCP/IP protocol layers) of the devices and permits data to be transferred directly, between the two devices, using certain memory buffers, thereby avoiding memory copies taking place when the existing network protocol layers are used.

## Claims

### Claim 1

A method of processing an incoming packet by a target device associated with a distributed storage system, the method comprising:
receiving the incoming packet from an initiator device, the incoming packet encapsulated by the initiator device using an Internet Small Computer Systems Interface (iSCSI) extension for remote direct memory access (RDMA) (iSER) of the initiator device;
decapsulating the packet to access a Small Computer Systems Interface (SCSI) read command indicating an input/output (I/O) operation to perform at the distributed storage system;
allocating a data structure in a memory of the target device for storing data received from the distributed storage system as a result of performance of the I/O operation;
storing the data in the data structure;
encapsulating the data in the data structure to generate a SCSI DATA-IN PDU including the data, wherein the SCSI DATA-IN PDU is stored in the data structure;
encapsulating the SCSI DATA-IN PDU to generate an outgoing iSCSI protocol data unit (PDU) including the SCSI DATA-IN PDU, wherein the outgoing iSCSI PDU is stored in the data structure;
encapsulating the iSCSI PDU to generate an outgoing iSER packet; and
transmitting the iSER packet to the initiator device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the decapsulating comprises removing an iSER header from the incoming packet.

### Claim 3 (depends on 1)

The method of claim 1, wherein the data structure comprises a scatter-gather list.

### Claim 4 (depends on 1)

The method of claim 1, wherein encapsulating the data in the data structure to generate a SCSI DATA-IN PDU is performed by a SCSI protocol layer of a network-storage stack of the target device, the method further comprising:
notifying, at the SCSI protocol layer, a iSCSI protocol layer of the network-storage stack about a memory location of the data structure, after encapsulating the data in the data structure to generate the SCSI DATA-IN PDU.

### Claim 5 (depends on 1)

The method of claim 1, wherein encapsulating the data in the data structure to generate a iSCSI PDU is performed by a iSCSI protocol layer of a network-storage stack of the target device, the method further comprising:
notifying, at the iSCSI protocol layer, a iSER protocol layer of the network-storage stack about a memory location of the data structure, after encapsulating the data in the data structure to generate the iSCSI PDU.

### Claim 6

A target apparatus, comprising:
a non-transitory memory comprising executable instructions; and
a processor in data communication with the memory and configured to execute the instructions to cause the apparatus to perform a method, the method comprising:
receiving the incoming packet from an initiator apparatus, the incoming packet encapsulated by the initiator device using an Internet Small Computer Systems Interface (iSCSI) extension for remote direct memory access (RDMA) (iSER) of the initiator apparatus;
decapsulating the packet to access a Small Computer Systems Interface (SCSI) read command indicating an input/output (I/O) operation to perform at the distributed storage system;
allocating a data structure in a memory of the target apparatus for storing data received from the distributed storage system as a result of performance of the I/O operation;
storing the data in the data structure;
encapsulating the data in the data structure to generate a SCSI DATA-IN PDU including the data, wherein the SCSI DATA-IN PDU is stored in the data structure;
encapsulating the SCSI DATA-IN PDU to generate an outgoing iSCSI protocol data unit (PDU) including the SCSI DATA-IN PDU, wherein the outgoing iSCSI PDU is stored in the data structure;
encapsulating the iSCSI PDU to generate an outgoing iSER packet; and
transmitting the iSER packet to the initiator apparatus.

### Claim 7 (depends on 6)

The target apparatus of claim 6, wherein the decapsulating comprises removing an iSER header from the incoming packet.

### Claim 8 (depends on 6)

The target apparatus of claim 6, wherein the data structure comprises a scatter-gather list.

### Claim 9 (depends on 6)

The target apparatus of claim 6, wherein encapsulating the data in the data structure to generate a SCSI DATA-IN PDU is performed by a SCSI protocol layer of a network-storage stack of the target apparatus, wherein the method further comprises:
notifying, at the SCSI protocol layer, a iSCSI protocol layer of the network-storage stack about a memory location of the data structure, after encapsulating the data in the data structure to generate the SCSI DATA-IN PDU.

### Claim 10 (depends on 6)

The target apparatus of claim 6, wherein encapsulating the data in the data structure to generate a iSCSI PDU is performed by a iSCSI protocol layer of a network-storage stack of the target device, wherein the method further comprises:
notifying, at the iSCSI protocol layer, a iSER protocol layer of the network-storage stack about a memory location of the data structure, after encapsulating the data in the data structure to generate the iSCSI PDU.

### Claim 11

A non-transitory computer readable medium having instructions stored thereon that, when executed by a target device, cause the target device to perform a method comprising:
receiving the incoming packet from an initiator device, the incoming packet encapsulated by the initiator device using an Internet Small Computer Systems Interface (iSCSI) extension for remote direct memory access (RDMA) (iSER) of the initiator device;
decapsulating the packet to access a Small Computer Systems Interface (SCSI) read command indicating an input/output (I/O) operation to perform at the distributed storage system;
allocating a data structure in a memory of the target device for storing data received from the distributed storage system as a result of performance of the I/O operation;
storing the data in the data structure;
encapsulating the data in the data structure to generate a SCSI DATA-IN PDU including the data, wherein the SCSI DATA-IN PDU is stored in the data structure;
encapsulating the SCSI DATA-IN PDU to generate an outgoing iSCSI protocol data unit (PDU) including the SCSI DATA-IN PDU, wherein the outgoing iSCSI PDU is stored in the data structure;
encapsulating the iSCSI PDU to generate an outgoing iSER packet; and
transmitting the iSER packet to the initiator device.

### Claim 12 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the decapsulating comprises removing an iSER header from the incoming packet.

### Claim 13 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the data structure comprises a scatter-gather list.

### Claim 14 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein encapsulating the data in the data structure to generate a SCSI DATA-IN PDU is performed by a SCSI protocol layer of a network-storage stack of the target apparatus, wherein the method further comprises:
notifying, at the SCSI protocol layer, a iSCSI protocol layer of the network-storage stack about a memory location of the data structure, after encapsulating the data in the data structure to generate the SCSI DATA-IN PDU.

### Claim 15 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein encapsulating the data in the data structure to generate a iSCSI PDU is performed by a iSCSI protocol layer of a network-storage stack of the target device, wherein the method further comprises:
notifying, at the iSCSI protocol layer, a iSER protocol layer of the network-storage stack about a memory location of the data structure, after encapsulating the data in the data structure to generate the iSCSI PDU.

### Claim 16

A method of processing an incoming packet by a target device associated with a distributed storage system, the method comprising:
receiving the incoming packet from an initiator device, the incoming packet encapsulated by the initiator device using an Internet Small Computer Systems Interface (iSCSI) extension for remote direct memory access (RDMA) (iSER) of the initiator device;
identifying a remote I/O buffer offset in an iSER header of the packet, the remote I/O buffer offset being associated with a remote I/O buffer at the initiator device, the remote I/O buffer including data associated with a SCSI write command in the incoming packet;
storing the remote I/O buffer in a memory of the target device;
decapsulating, at the iSER of the target device, the packet to access an Internet Small Computer Systems Interface (iSCSI) packet data unit (PDU);
decapsulating the iSCSI PDU to access the SCSI write command;
allocating a data structure in the memory for storing the data associated with the SCSI write command;
performing an RDMA operation to read the data from the remote I/O buffer of the initiator device;
storing the read data into the data structure; and
causing a write operation associated with the SCSI write command to be performed by the distributed storage system, wherein the distributed storage system accesses the data structure to perform the write operation using the data in the data structure.
