# US 11,444,886 B1 · Out of order packet buffer selection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,444,886 B1
- Document ID: US11444886B1
- Status: Granted
- Kind: B1
- Application number: 16/577,625
- Filed: 2019-09-20
- Granted: 2022-09-13
- Earliest claimed: 2018-09-21
- Assignees: Marvell Asia Pte Ltd
- Applicants: Marvell Asia Pte Ltd
- Inventors: Noa Stawitzky; Yoav Galon; Amit Radzi; Shaham Kenat; Michael Shteinbok
- Web page: https://patentgrep.com/patent/US11444886B1

## CPC

- H04L 47/34: ensuring sequence integrity, e.g. using sequence numbers
- G06F 13/28: using burst mode transfer, e.g. direct memory access {DMA}, cycle steal (G06F13/32 takes precedence)
- H04L 47/193: at the transport layer, e.g. TCP related
- H04L 47/30: in combination with information about buffer occupancy at either end or at transit nodes
- H04L 47/32: by discarding or delaying data units, e.g. packets or frames
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for storing a packet in an application buffer of a processor node is described. A first out of order (OOO) packet within a stream of packets transmitted from a second network interface controller is received in a sequence represented by respective sequence numbers that are assigned to corresponding packets in the stream. The first OOO packet is received out of order relative to at least a first missing packet of the stream of packets. One of an OOO buffer associated with the first network interface controller or the application buffer associated with the first network interface controller is selected. The OOO buffer is configured to store OOO packets. The first network interface controller is configured to store the stream of packets in sequential order in the application buffer. The first OOO packet is stored in the selected buffer.

## Claims

### Claim 1

A method for storing a packet in an application buffer of a processor node, the method comprising:
receiving, at a first network interface controller of the processor node, a first out of order (OOO) packet within a stream of packets transmitted from a second network interface controller in a sequence represented by respective sequence numbers that are assigned to corresponding packets in the stream, the first OOO packet being received out of order relative to at least a first missing packet of the stream of packets;
selecting, at the first network interface controller, one buffer from among multiple buffers for storing the first OOO packet, the multiple buffers including an OOO buffer associated with the first network interface controller and the application buffer associated with the first network interface controller, wherein the OOO buffer is configured to store OOO packets, and wherein the first network interface controller is configured to store the stream of packets in sequential order in the application buffer;
in response to the first network interface controller selecting the application buffer for storing the first OOO packet, storing the first OOO packet in the application buffer; and
in response to the first network interface controller selecting the OOO buffer for storing the first OOO packet:
storing, by the first network interface controller, the first OOO packet in the OOO buffer, and
after receiving the missing packet, copying the first OOO packet from the OOO buffer to the application buffer.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first OOO packet i) has a sequence number that is larger than a sequence number of the first missing packet, and ii) is received before the first missing packet.

### Claim 3 (depends on 1)

The method of claim 1, wherein selecting the one buffer from among the multiple buffers comprises:
selecting the application buffer in response to determining that first out of order placement (OOP) information corresponding to the first OOO packet can be stored in an OOP information buffer that is configured to store OOP information corresponding to OOO packets stored in the application buffer; and
selecting the OOO buffer in response to determining that first OOP information that corresponds to the first OOO packet cannot be stored in the OOP information buffer.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
when the first network interface controller selects the application buffer for storing the first OOO packet, storing the first OOO packet in the application buffer at a location that is offset from an end of a most recently received in-order packet of the stream of packets by an offset amount corresponding to the first missing packet, and storing the first OOP information in the OOP information buffer.

### Claim 5 (depends on 4)

The method of claim 4, the method further comprising:
receiving, at the first network interface controller, the first missing packet;
storing, by the first network interface controller, the first missing packet in the application buffer at a location that is after a most recently received in-order packet of the stream of packets;
determining, at the first network interface controller, whether the offset amount within the application buffer has been filled by the first missing packet;
when the first missing packet has not filled the offset amount within the application buffer, sending a completion indicator that corresponds to the first missing packet to the second network interface controller;
when the first missing packet has filled the offset amount within the application buffer and when one or more OOO packets of the stream of packets are stored in the OOO buffer, copying the one or more OOO packets from the OOO buffer to the application buffer.

### Claim 6 (depends on 5)

The method of claim 5, further comprising sending a completion indicator to the second network interface controller that corresponds to i) the first missing packet, ii) the first OOO packet, and iii) when the one or more OOO packets have been copied from the OOO buffer to the application buffer, the OOO packets copied to the application buffer.

### Claim 7 (depends on 3)

The method of claim 3, wherein selecting the one buffer from among the multiple buffers comprises selecting the one buffer from among the multiple buffers based on a determination of which remote direct memory access (RDMA) operation, among a plurality of different RDMA operations, is included within the first OOO packet.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
determining, at the first network interface controller, whether the determined RDMA operation within the first OOO packet allows for out of order placement of the first OOO packet in the application buffer; and
selecting the application buffer when it is determined that the OOP information buffer has available space and the determined RDMA operation within the first OOO packet allows for out of order placement.

### Claim 9 (depends on 3)

The method of claim 3, further comprising designating the OOP information buffer for a single packet stream of a plurality of packet streams received at the first network interface controller.

### Claim 10 (depends on 3)

The method of claim 3, wherein the method further comprises dropping the first OOO packet when i) the first network interface controller selects the OOO buffer for storing the first OOO packet and ii) the OOO buffer is full.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
when the first network interface controller selects the application buffer for storing the first OOO packet, storing the first OOO packet in the application buffer using a direct memory access (DMA) bus of a processor node that contains the application buffer.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
determining, at the first network interface controller, whether an out of order placement (OOP) information buffer has available space to store first OOP information corresponding to the first OOO packet, the OOP information buffer being configured to store OOP information corresponding to OOO packets stored in the application buffer;
wherein selecting the one buffer from among the multiple buffers comprises:
selecting the application buffer to store the first OOO packet in response to determining that the OOP information buffer has available space to store the first OOP information corresponding to the first OOO packet, and
selecting the OOO buffer to store the first OOO packet in response to determining that the OOP information buffer does not have available space to store the first OOP information corresponding to the first OOO packet.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
determining, at the first network interface controller, which remote direct memory access (RDMA) operation, among a plurality of different RDMA operations, is included within the first OOO packet;
wherein selecting the one buffer from among the multiple buffers comprises:
selecting the application buffer to store the first OOO packet in response to determining that i) the OOP information buffer has available space, and ii) the determined RDMA operation is a first type of RDMA operation, and
selecting the OOO buffer to store the first OOO packet in response to determining that the determined RDMA operation is a second type of RDMA operation.

### Claim 14

A network interface controller for storing a packet in an application buffer of a first processor node, the network interface controller comprising:
a processor configured to:
access the application buffer of the processor node, the application buffer being configured to store a stream of packets in sequential order,
access an out of order placement (OOP) information buffer configured to store OOP information corresponding to OOO packets stored in the application buffer, and
access an out of order (OOO) buffer configured to store OOO packets,
receive a first OOO packet within the stream of packets, the stream of packets being transmitted from a second processor node in a sequence represented by respective sequence numbers that are assigned to corresponding packets in the stream, the first OOO packet being received out of order relative to at least a first missing packet of the stream of packets,
select one buffer from among multiple buffers for storing the first OOO packet, the multiple buffers including the OOO buffer and the application buffer,
in response to selecting the application buffer for storing the first OOO packet, store the first OOO packet in the application buffer, and
in response to selecting the OOO buffer for storing the first OOO packet,
store the first OOO packet in the OOO buffer, and
after receiving the missing packet, copy the first OOO packet from the OOO buffer to the application buffer.

### Claim 15 (depends on 14)

The network interface controller of claim 14, wherein the first OOO packet i) has a sequence number that is larger than a sequence number of the first missing packet, and ii) is received before the first missing packet.

### Claim 16 (depends on 14)

The network interface controller of claim 14, wherein the processor is further configured to:
select the application buffer in response to determining that first OOP information corresponding to the first OOO packet can be stored in the OOP information buffer; and
select the OOO buffer in response to determining that first OOP information that corresponds to the first OOO packet cannot be stored in the OOP information buffer.

### Claim 17 (depends on 16)

The network interface controller of claim 16, wherein the processor is further configured to:
when the processor selects the application buffer, store the first OOO packet in the application buffer at a location that is offset from an end of a most recently received in-order packet of the stream of packets by an offset amount corresponding to the first missing packet, and store the first OOP information in the OOP information buffer.

### Claim 18 (depends on 17)

The network interface controller of claim 17, wherein the processor is further configured to:
receive the first missing packet;
store the first missing packet in the application buffer at a location that is after a most recently received in-order packet of the stream of packets;
determine whether the offset amount within the application buffer has been filled by the first missing packet;
when the first missing packet has not filled the offset amount within the application buffer, send a completion indicator that corresponds to the first missing packet to the second processor node;
when the first missing packet has filled the offset amount within the application buffer and when one or more OOO packets of the stream of packets are stored in the OOO buffer,
copy the one or more OOO packets from the OOO buffer to the application buffer.

### Claim 19 (depends on 18)

The network interface controller of claim 18, wherein the processor is further configured to send a completion indicator to the second processor node that corresponds to i) the first missing packet, ii) the first OOO packet, and iii) when the one or more OOO packets have been copied from the OOO buffer to the application buffer, the OOO packets copied to the application buffer.

### Claim 20 (depends on 18)

The network interface controller of claim 18, wherein the processor is further configured to, when the processor selects the application buffer for storing the first OOO packet, store the first OOO packet in the application buffer using a direct memory access (DMA) bus of the first processor node.

### Claim 21 (depends on 20)

The network interface controller of claim 20, wherein the application buffer is configured to store data at the application layer of the Open Systems Interconnection model.

### Claim 22 (depends on 16)

The network interface controller of claim 16, wherein the processor is configured to select the one buffer from among the multiple buffers based on a determination of which remote direct memory access (RDMA) operation, among a plurality of different RDMA operations, is included within the first OOO packet.

### Claim 23 (depends on 22)

The network interface controller of claim 22, wherein the processor is configured to determine at least one of i) whether the OOP information buffer has available space to store the first OOP information, and ii) whether the determined RDMA operation within the first OOO packet allows for out of order placement of the first OOO packet in the application buffer;
wherein the buffer selector is configured to select the application buffer when it is determined that the OOP information buffer has available space and the determined RDMA operation within the first OOO packet allows for out of order placement.

### Claim 24 (depends on 16)

The network interface controller of claim 16, wherein the processor is configured to designate the OOP information buffer for a single packet stream of a plurality of packet streams received at the first network interface controller.

### Claim 25 (depends on 16)

The network interface controller of claim 16, wherein the processor is further configured to drop the OOO packet when the OOO buffer is the selected buffer and the OOO buffer is full.

### Claim 26 (depends on 14)

The network interface controller of claim 14, wherein the network interface controller comprises the OOP information buffer and the processor node comprises the OOO buffer.

### Claim 27 (depends on 14)

The network interface controller of claim 14, wherein the processor node comprises the OOP information buffer and the OOO buffer.

### Claim 28 (depends on 14)

The network interface controller of claim 14, wherein the processor is further configured to:
determine whether the OOP information buffer has available space to store first OOP information corresponding to the first OOO packet;
select the application buffer to store the first OOO packet in response to determining that the OOP information buffer has available space to store the first OOP information corresponding to the first OOO packet; and
select the OOO buffer to store the first OOO packet in response to determining that the OOP information buffer does not have available space to store the first OOP information corresponding to the first OOO packet.

### Claim 29 (depends on 28)

The network interface controller of claim 28, wherein the processor is further configured to:
determine which remote direct memory access (RDMA) operation, among a plurality of different RDMA operations, is included within the first OOO packet;
select the application buffer to store the first OOO packet in response to determining that i) the OOP information buffer has available space, and ii) the determined RDMA operation is a first type of RDMA operation; and
select the OOO buffer to store the first OOO packet in response to determining that the determined RDMA operation is a second type of RDMA operation.