# US 11,115,343 B2 · Transport layer providing deterministic transport across multiple deterministic data links > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,115,343 B2 - Document ID: US11115343B2 - Status: Granted - Kind: B2 - Application number: 15/725,502 - Filed: 2017-10-05 - Published: 2019-04-11 - Granted: 2021-09-07 - Earliest claimed: 2017-10-05 - Assignees: Cisco Technology, Inc. - Applicants: Cisco Technology, Inc. - Inventors: Pascal Thubert; Patrick Wetterwald; Eric Michel Levy-Abegnoli - Web page: https://patentgrep.com/patent/US11115343B2 ## CPC - H04L 69/326: in the transport layer [OSI layer 4] (H04L69/16 takes precedence) - H04L 47/193: at the transport layer, e.g. TCP related - H04L 47/34: ensuring sequence integrity, e.g. using sequence numbers - H04L 49/35: Switches specially adapted for specific applications - H04L 49/90: Buffering arrangements - H04L 67/5681: Pre-fetching or pre-delivering data based on network characteristics - H04L 67/60: Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources (admission control or resource allocation H04L47/70) - H04L 69/16: Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract In one embodiment, a method comprises: receiving, by a transport layer executed by a processor circuit in an apparatus, a flow of application data having been originated by an executable application; storing, by the transport layer, the application data as transport layer packets in a buffer circuit in the apparatus, each transport layer packet having a corresponding transport sequence identifier identifying a corresponding position of the transport layer packet relative to a transmit order of the transport layer packets; and causing, by the transport layer, a plurality of deterministic network interface circuits to deterministically retrieve the transport layer packets, in the transmit order, from the buffer circuit for deterministic transmission across respective deterministic links, the transport sequence identifiers enabling a destination transport layer to recover the transmit order of the transport layer following the deterministic transmission across the deterministic links, regardless of order of reception by the destination transport layer. ## Claims ### Claim 1 A method comprising: receiving, by a transport layer executed by a processor circuit in an apparatus, an identified flow of application data having been originated by an executable application; storing, by the transport layer, the identified flow of application data as transport layer packets in a buffer circuit in the apparatus, each transport layer packet having a corresponding transport sequence identifier identifying a corresponding position of the transport layer packet relative to a transmit order of the transport layer packets; and causing, by the transport layer, a plurality of deterministic network interface circuits to deterministically retrieve the transport layer packets, in the transmit order, from the buffer circuit for deterministic transmission across respective deterministic links according to deterministic constraints required for the identified flow of application data, the transport sequence identifiers enabling a destination transport layer to recover the transmit order of the transport layer following the deterministic transmission across the deterministic links according to the deterministic constraints, regardless of order of reception thereof by the destination transport layer, the deterministic transmission distinct from non-deterministic communications by the apparatus, the deterministic constraints requiring timing synchronization and scheduled transmissions along each hop of the deterministic links from the deterministic network interface circuits to one or more destination deterministic network interface circuits of a destination device executing the destination transport layer. ### Claim 2 (depends on 1) The method of claim 1, wherein: the receiving includes receiving the identified flow of application data as a sequence of portions of the application data from the executable application executed locally by the processor circuit; the storing including the transport layer generating the transport layer packets each comprising one or more portions of the application data, and generating and inserting the corresponding transport sequence identifier into each corresponding transport layer packet. ### Claim 3 (depends on 1) The method of claim 1, wherein: the receiving includes receiving the identified flow of application data within received transport layer packets, in a received order, across a second distinct plurality of deterministic network interface circuits having respective deterministic links for receiving the transport layer packets; the method further comprising reordering the received transport layer packets from the received order into the transmit order based on the respective transport sequence identifiers within the received transport layer packets. ### Claim 4 (depends on 3) The method of claim 3, further comprising the transport layer allocating a receive buffer, for storage of the received transport layer packets, based on a maximum determined latency across the second plurality of deterministic network interface circuits, a maximum number of the received transport layer packets transmittable within the maximum determined latency, and a maximum size of the received transport layer packets. ### Claim 5 (depends on 1) The method of claim 1, further comprising the transport layer allocating the buffer circuit, for storage of the transport layer packets, based on a maximum determined latency across the plurality of deterministic network interface circuits, a maximum number of the transport layer packets transmittable within the maximum determined latency, and a maximum size of the transport layer packets. ### Claim 6 (depends on 1) The method of claim 1, wherein the causing includes: determining, by the transport layer, a single one of the deterministic links has insufficient resources for deterministic transport of the transport layer packets to the destination transport layer; and sending, by the transport layer, a request to a device manager, executed in the apparatus, to allocate the deterministic network interface circuits for the deterministic transport of the transport layer packets across the deterministic links. ### Claim 7 (depends on 1) The method of claim 1, wherein the causing includes: causing each deterministic network interface to deterministically retrieve each transport layer packet from the buffer circuit by a pre-fetch interval preceding a corresponding deterministic transmission, the pre-fetch interval including a retrieval latency for the corresponding deterministic network interface to retrieve the corresponding transport layer packet from the buffer circuit, and the pre-fetch interval further including a transmit queuing latency for the corresponding deterministic network interface to queue the corresponding transport layer packet for the corresponding deterministic transmission. ### Claim 8 (depends on 7) The method of claim 7, wherein the deterministic network interface circuits are one of: device interface circuits coupled to the apparatus and configured for accessing the buffer circuit via a local bus; or a deterministic switching device configured for accessing the buffer circuit via a top-of-rack switch. ### Claim 9 (depends on 1) The method of claim 1, wherein the causing includes: receiving, by the transport layer from one of the deterministic network interface circuits, a message specifying one of the transport sequence identifiers indicating successful deterministic transmission of the corresponding transport layer packet; discarding from the buffer circuit, by the transport layer in response to the message, the corresponding transport layer packet having the one transport sequence identifier. ### Claim 10 An apparatus comprising: a memory circuit; a data bus; and a processor circuit configured for executing a transport layer, the transport layer configured for receiving an identified flow of application data having been originated by an executable application; the transport layer further configured for storing the identified flow of application data as transport layer packets in the memory circuit, each transport layer packet having a corresponding transport sequence identifier identifying a corresponding position of the transport layer packet relative to a transmit order of the transport layer packets; the transport layer further configured for causing a plurality of deterministic network interface circuits to deterministically retrieve the transport layer packets, in the transmit order, from the memory circuit via the data bus for deterministic transmission across respective deterministic links according to deterministic constraints required for the identified flow of application data, the transport sequence identifiers enabling a destination transport layer to recover the transmit order of the transport layer following the deterministic transmission across the deterministic links according to the deterministic constraints, regardless of order of reception thereof by the destination transport layer, the deterministic transmission distinct from non-deterministic communications by the apparatus, the deterministic constraints requiring timing synchronization and scheduled transmissions along each hop of the deterministic links from the deterministic network interface circuits to one or more destination deterministic network interface circuits of a destination device executing the destination transport layer. ### Claim 11 (depends on 10) The apparatus of claim 10, wherein: the processor circuit further is configured for executing an executable application that generates the identified flow of application data as a sequence of portions of the application data; the transport layer configured for receiving the sequence of portions of the application data; the transport layer configured for generating the transport layer packets each comprising one or more portions of the application data, the transport layer further configured for generating and inserting the corresponding transport sequence identifier into each corresponding transport layer packet. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein: the transport layer is configured for receiving the identified flow of application data within a plurality of received transport layer packets, in a received order, across a second distinct plurality of deterministic network interface circuits having respective deterministic links for receiving the transport layer packets; the transport layer configured for reordering the received transport layer packets from the received order into the transmit order based on the respective transport sequence identifiers within the received transport layer packets. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the transport layer is configured for allocating in the memory circuit a receive buffer, for storage of the received transport layer packets, based on a maximum determined latency across the second plurality of deterministic network interface circuits, a maximum number of the received transport layer packets transmittable within the maximum determined latency, and a maximum size of the received transport layer packets. ### Claim 14 (depends on 10) The apparatus of claim 10, wherein the transport layer is configured for allocating a transmit buffer in the memory circuit, for storage of the transport layer packets, based on a maximum determined latency across the plurality of deterministic network interface circuits, a maximum number of the transport layer packets transmittable within the maximum determined latency, and a maximum size of the transport layer packets. ### Claim 15 (depends on 10) The apparatus of claim 10, further comprising a device manager configured for allocating the deterministic network interface circuits for deterministic transmissions, wherein the transport layer is configured for: determining a single one of the deterministic links has insufficient resources for deterministic transport of the transport layer packets to the destination transport layer; and sending a request to a device manager to allocate the deterministic network interface circuits for the deterministic transport of the transport layer packets across the deterministic links; the device manager configured for sending a request to a network manager for allocation of the deterministic links for the deterministic transmissions of the transport layer packets. ### Claim 16 (depends on 10) The apparatus of claim 10, wherein the transport layer is configured for: causing each deterministic network interface circuit to deterministically retrieve each transport layer packet from the memory circuit by a pre-fetch interval preceding a corresponding deterministic transmission, the pre-fetch interval including a retrieval latency for the corresponding deterministic network interface circuit to retrieve the corresponding transport layer packet from the memory circuit, and the pre-fetch interval further including a transmit queuing latency for the corresponding deterministic network interface to queue the corresponding transport layer packet for the corresponding deterministic transmission. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein: the apparatus further comprises the deterministic network interface circuits, each configured for accessing the memory circuit via the data bus; or the apparatus further comprises a device interface circuit coupled to the data bus and configured for communicating with each deterministic network interface circuit via a top-of-rack switch. ### Claim 18 (depends on 10) The apparatus of claim 10, wherein the transport layer is configured for: receiving, from one of the deterministic network interface circuits, a message specifying one of the transport sequence identifiers indicating successful deterministic transmission of the corresponding transport layer packet; discarding from the memory circuit, in response to the message, the corresponding transport layer packet having the one transport sequence identifier. ### Claim 19 One or more non-transitory tangible media encoded with logic for execution by a machine and when executed by the machine operable for: receiving, by a transport layer executed by the machine, an identified flow of application data having been originated by an executable application; storing, by the transport layer, the identified flow of application data as transport layer packets in a buffer circuit in the machine, each transport layer packet having a corresponding transport sequence identifier identifying a corresponding position of the transport layer packet relative to a transmit order of the transport layer packets; and causing, by the transport layer, a plurality of deterministic network interface circuits to deterministically retrieve the transport layer packets, in the transmit order, from the buffer circuit for deterministic transmission across respective deterministic links according to deterministic constraints required for the identified flow of application data, the transport sequence identifiers enabling a destination transport layer to recover the transmit order of the transport layer following the deterministic transmission across the deterministic links according to the deterministic constraints, regardless of order of reception thereof by the destination transport layer, the deterministic transmission distinct from non-deterministic communications by the machine, the deterministic constraints requiring timing synchronization and scheduled transmissions along each hop of the deterministic links from the deterministic network interface circuits to one or more destination deterministic network interface circuits of a destination device executing the destination transport layer. ### Claim 20 (depends on 19) The one or more non-transitory tangible media of claim 19, wherein the causing includes: determining, by the transport layer, a single one of the deterministic links has insufficient resources for deterministic transport of the transport layer packets to the destination transport layer; and sending, by the transport layer, a request to a device manager to allocate the deterministic network interface circuits for the deterministic transport of the transport layer packets across the deterministic links.