# US 11,451,476 B2 · Multi-path transport design

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,451,476 B2
- Document ID: US11451476B2
- Status: Granted
- Kind: B2
- Application number: 16/539,303
- Filed: 2019-08-13
- Published: 2019-11-28
- Granted: 2022-09-20
- Earliest claimed: 2015-12-28
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Leah Shalev; Nafea Bshara; Georgy Machulsky; Brian William Barrett
- Web page: https://patentgrep.com/patent/US11451476B2

## CPC

- H04L 69/10: Streamlined, light-weight or high-speed protocols, e.g. express transfer protocol [XTP] or byte stream
- G06F 9/4494: {data driven}
- G06F 9/5005: {to service a request}
- H04L 43/0823: Errors, e.g. transmission errors
- H04L 43/0852: Delays
- H04L 45/24: Multipath
- H04L 45/72: {Routing based on the source address}
- H04L 45/74: Address processing for routing
- H04L 47/125: by balancing the load, e.g. traffic engineering
- H04L 49/25: Routing or path finding in a switch fabric
- H04L 61/5007: Internet protocol [IP] addresses
- H04L 65/40: Support for services or applications
- H04L 69/16: Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L 69/22: Parsing or analysis of headers
- G06F 8/36: Software reuse
- G06F 9/4411: {Configuring for operating with peripheral devices; Loading of device drivers}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Disclosed herein is a method including receiving, from a user application, data to be transmitted from a source address to a destination address using a single connection through a network; and splitting the data into a plurality of packets according to a communication protocol. For each packet of the plurality of packets, a respective flowlet for the packet to be transmitted in is determined from a plurality of flowlets; a field in the packet used by a network switch of the network to route the packet is set based on the determined flowlet for the packet; and the packet is sent via the determined flowlet for transmitting through the network.

## Claims

### Claim 1

A method comprising:
receiving, at a network adapter of a first network endpoint device, a data stream from a user application executing on the first network endpoint device, the network adapter having one or more physical ports in communication with a network, wherein the data stream is to be transmitted to a second network endpoint device, wherein the first network endpoint device is associated with a source address on the network, wherein the second network endpoint device is associated with a destination address on the network;
splitting the data stream into packets according to a transport service of a transport layer, wherein splitting the data stream into the packets includes generating headers for the packets;
assigning a set of the packets to a particular flowlet from a plurality of flowlets maintained by the transport layer, wherein each flowlet of the plurality of flowlets uses a different path over the network when transmitting the respective packets assigned to the flowlet, wherein all the respective packets assigned to the flowlet use a same path over the network between the first network endpoint device and the second network endpoint device, and wherein a number of flowlets in use is dynamically adjusted when assigning the set of the packets to the particular flowlet;
modifying a field in headers of the set of the packets, wherein the field is used to route the set of the packets over the network, wherein modification of the field causes each of the set of the packets to be routed according to a path over the network associated with the particular flowlet; and
sending, using the one or more physical ports, the set of the packets over the network according to the transport service.

### Claim 2 (depends on 1)

The method of claim 1, wherein the field being modified is a protocol optional field.

### Claim 3 (depends on 2)

The method of claim 2, wherein the protocol optional field is a source port field.

### Claim 4 (depends on 1)

The method of claim 1, wherein the packets being transmitted on different paths have the same source address, destination address, and destination port.

### Claim 5 (depends on 1)

The method of claim 1, wherein each packet of the set of the packets includes at least one of a flowlet index or a packet sequence number.

### Claim 6 (depends on 5)

The method of claim 5, wherein the second network endpoint device obtains the flowlet index or the packet sequence number of a received packet from a header of the received packet.

### Claim 7

An integrated circuit device comprising:
processing logic; and
a memory coupled to the processing logic and storing program code, which when executed by the processing logic, implements operations including:
receiving a message from a user application to be transmitted to a network endpoint device associated with a destination address on a network;
splitting the message into packets according to a transport service of a transport layer;
assigning a set of the packets to a particular flowlet from a plurality of flowlets maintained by the transport layer and dynamically adjusting a number of flowlets in use when assigning the set of the packets to the particular flowlet, wherein each flowlet of the plurality of flowlets uses a different path over the network when transmitting the respective packets assigned to the flowlet, and wherein all the respective packets assigned to the flowlet use a same path over the network to reach the network endpoint device;
modifying a field in headers of the set of the packets, wherein the field is used to route the set of the packets over the network, wherein modification of the field causes each of the set of the packets to be routed according to a path over the network associated with the particular flowlet; and
sending the set of the packets over the network according to the transport service.

### Claim 8 (depends on 7)

The integrated circuit device of claim 7, wherein the field being modified is a protocol optional field.

### Claim 9 (depends on 8)

The integrated circuit device of claim 8, wherein the protocol optional field is a source port field.

### Claim 10 (depends on 7)

The integrated circuit device of claim 7, wherein the packets being transmitted on different paths have the same source address, destination address, and destination port.

### Claim 11 (depends on 7)

The integrated circuit device of claim 7, wherein the operations further include keeping a number of active packets in each of the plurality of flowlets below a threshold.

### Claim 12 (depends on 11)

The integrated circuit device of claim 11, wherein the threshold is adjustable based on network congestion.

### Claim 13

A computing device comprising:
a processor;
a memory coupled to the processor and storing a user application; and
a network adapter coupled to the processor and having one or more ports in communication with a network, wherein the network adapter is operable to:
receive data from the user application to be transmitted to a network endpoint device associated with a destination address on the network;
split the data into packets according to a transport service of a transport layer;
assign a set of the packets to a particular flowlet from a plurality of flowlets maintained by the transport layer and dynamically adjust a number of flowlets in use when assigning the set of the packets to the particular flowlet, wherein each flowlet of the plurality of flowlets uses a different path over the network when transmitting the respective packets assigned to the flowlet, and wherein all the respective packets assigned to the flowlet use a same path over the network to reach the network endpoint device;
modify a field in headers of the set of the packets, wherein the field is used to route the set of the packets over the network, wherein modification of the field causes each of the set of the packets to be routed according to a path over the network associated with the particular flowlet; and
send the set of the packets over the network according to the transport service.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein the field being modified is a protocol optional field.

### Claim 15 (depends on 14)

The computing device of claim 14, wherein the protocol optional field is a source port field.

### Claim 16 (depends on 13)

The computing device of claim 13, wherein the packets being transmitted on different paths have the same source address, destination address, and destination port.

### Claim 17 (depends on 13)

The computing device of claim 13, wherein the network adapter is further operable to select the particular flowlet from among the plurality of flowlets based on at least one of:
a number of active packets in each flowlet of the plurality of flowlets;
a latency of each flowlet of the plurality of flowlets;
a drop rate of each flowlet of the plurality of flowlets;
information provided by the user application; or
an indication from the user application regarding how the packets are to be assigned to the plurality of flowlets.

### Claim 18 (depends on 13)

The computing device of claim 13, wherein each packet of the set of the packets includes at least one of a flowlet index or a packet sequence number.

### Claim 19 (depends on 13)

The computing device of claim 13, wherein the network adapter is implemented using a system-on-chip (SOC), a network interface card (NIC), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA).