# US 9,882,771 B2 · Completion tracking for groups of transfer requests

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,882,771 B2
- Document ID: US9882771B2
- Status: Granted
- Kind: B2
- Application number: 14/485,958
- Filed: 2014-09-15
- Published: 2015-01-01
- Granted: 2018-01-30
- Earliest claimed: 2009-06-30
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Rabin A. Sugumar; Bjørn Dag Johnsen; Lars Paul Huse
- Web page: https://patentgrep.com/patent/US9882771B2

## CPC

- H04L 47/10: Flow control; Congestion control
- H04L 41/065: {involving logical or physical relationship, e.g. grouping and hierarchies}
- H04L 47/26: using explicit feedback to the source, e.g. choke packets
- H04L 47/283: in response to processing delays, e.g. caused by jitter or round trip time [RTT]
- H04L 47/30: in combination with information about buffer occupancy at either end or at transit nodes
- H04L 49/00: Packet switching elements
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Techniques for tracking completion of transfer requests. In one embodiment, a compute node connects to a network adapter (NA). In one embodiment, software running on the compute node contains instructions in which some remote data transfer requests belong to (or are associated with) completion groups. These completion groups may be constructed so that the system may more efficiently determine the completion status of remote transfer requests. In one embodiment, The NA includes a hardware counter for each completion group (CG). In one embodiment, the counter is configured to count when each transfer request in the completion group is received and when each request in the completion group is completed. For example, the counter may increment on receipt and decrement on completion such that a zero indicates completion of all requests in the completion group. In one embodiment, the NA includes a flush register to indicate whether the counter is valid.

## Claims

### Claim 1

An apparatus, comprising:
a host interface configured to receive a plurality of transfer requests from a local compute node, wherein requests of the plurality of transfer requests are associated with a completion group;
a network interface configured to transmit the plurality of transfer requests to one or more other compute nodes via a network; and
counter logic circuitry configured to:
count when each transfer request associated with the completion group is received; and
count when each transfer request associated with the completion group that was transmitted via the network is completed;
a flush indicator configured to indicate, based on reception of an ordering command from the local compute node, whether a current count of the counter logic circuitry is valid, wherein the ordering command forces outstanding transfer requests from the local compute node to the network interface ahead of the ordering command;
wherein the apparatus is configured to indicate, to the local compute node in response to a command from the local compute node via the host interface, a number of uncompleted requests in the completion group based on the counter logic circuitry.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein, to count when each transfer request associated with the completion group is received, the counter logic circuitry is configured to increment a counter and wherein, to count when each transfer request associated with the completion group that was transmitted via the network is completed, the counter logic circuitry is configured to decrement the counter.

### Claim 3 (depends on 1)

The apparatus of claim 1, further comprising:
second counter logic circuitry configured to:
count when each transfer request associated with a second completion group is received via the host interface; and
count when each transfer request associated with the second completion group that was transmitted via the network is completed;
wherein the apparatus is configured to indicate, to the local compute node, a number of uncompleted requests in the second completion group based on the second counter logic circuitry.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the apparatus is configured to clear the flush indicator and reset the counter logic circuitry in response to receiving a flush reset operation from the local compute node.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the apparatus is configured to utilize the counter logic circuitry to count transfer requests associated with a second completion group after completion of all requests in the completion group.

### Claim 6

A method, comprising:
receiving, via a local interface, a plurality of transfer requests from a local compute node, wherein requests of the plurality of transfer requests are associated with a completion group;
counting, using counter circuitry, when each transfer request associated with the completion group is received;
transmitting, on a network interface, the plurality of transfer requests;
counting, using the counter circuitry, when each transmitted transfer request is completed;
indicating, based on reception of an ordering command from the local compute node, whether a current count of the counter circuitry is valid, wherein the ordering command forces outstanding transfer requests from the local compute node to the network interface ahead of the ordering command; and
determining, based on the counting the reception and completion of the plurality of transfer requests and the indication, completion status of the completion group.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
assigning, by the local compute node, the plurality of transfer requests to the completion group;
sending, from the local compute node, the plurality of transfer requests via the local interface; and
receiving, by the local compute node, the completion status of the completion group.

### Claim 8 (depends on 6)

The method of claim 6, further comprising:
assigning the completion group to the counter circuitry; and
assigning a second completion group of a second plurality of transfer requests to second counter circuitry.

### Claim 9 (depends on 6)

The method of claim 6, further comprising:
sending, by the local compute node, the ordering command via the local interface;
polling a flush register, by the local compute node, until the flush register is set by the indicating; and
sending, by the local compute node after completion of the polling, a flush reset command.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
assigning the counter circuitry to a second completion group after sending the flush reset command.

### Claim 11 (depends on 6)

The method of claim 6, further comprising:
receiving, via the local interface, a flush reset command; and
resetting a flush register and the counter circuitry in response to the flush reset command.

### Claim 12

A system, comprising:
a network adapter comprising:
a local interface;
a network interface; and
first counter circuitry;
wherein the network adapter is configured to:
receive, via the local interface, a plurality of transfer requests associated with a first completion group;
count, using the first counter circuitry, a number of received transfer requests associated with the first completion group; and
count, using the first counter circuitry, a number of completed transfer requests associated with the first completion group
indicate, using a flush indicator based on reception of an ordering command from the local interface, whether a current count of the counter circuitry is valid, wherein the ordering command forces outstanding transfer requests from a local compute node to the network interface ahead of the ordering command.

### Claim 13 (depends on 12)

The system of claim 12, further comprising:
a compute node configured to:
assign the plurality of transfer requests to the first completion group and transmit the plurality of transfer requests via the local interface; and
determine completion status of the first completion group based on the first counter circuitry.

### Claim 14 (depends on 13)

The system of claim 13,
wherein the compute node is configured to, before determining the completion status, transmit the ordering command via the local interface and wait to determine the completion status until determining that the flush indicator is set.

### Claim 15 (depends on 14)

The system of claim 14, wherein the compute node is configured to send a flush reset command and assign the first counter circuitry to a second completion group after determining that the first completion group has completed.

### Claim 16 (depends on 12)

The system of claim 12, wherein the network adapter further comprises:
second counter circuitry;
wherein the network adapter is configured to:
receive, via the local interface, a plurality of transfer requests associated with a second completion group;
count, using the second counter circuitry, a number of received transfer requests associated with the second completion group; and
count, using the second counter circuitry, a number of completed transfer requests associated with the second completion group.