# US 10,771,401 B2 · Forwarding element data plane with computing parameter distributor > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,771,401 B2 - Document ID: US10771401B2 - Status: Granted - Kind: B2 - Application number: 16/147,754 - Filed: 2018-09-30 - Published: 2020-02-13 - Granted: 2020-09-08 - Earliest claimed: 2018-08-13 - Assignees: Barefoot Networks, Inc. - Applicants: Barefoot Networks, Inc. - Inventors: Masoud Moshref Javadi; Changhoon Kim; Patrick W. Bosshart; Anurag Agrawal - Web page: https://patentgrep.com/patent/US10771401B2 ## CPC - H04L 49/3063: {Pipelined operation} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 3/098: Distributed learning, e.g. federated learning - H04L 49/208: {Port mirroring} - H04L 49/9042: {Separate storage for different parts of the packet, e.g. header and payload} - G06F 2009/45595: {Network integration; Enabling network access in virtual machine instances} - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Some embodiments provide a network forwarding element with a data-plane forwarding circuit that has a parameter collecting circuit to store and distribute parameter values computed by several machines in a network. In some embodiments, the machines perform distributed computing operations, and the parameter values that compute are parameter values associated with the distributed computing operations. The parameter collecting circuit of the data-plane forwarding circuit (data plane) in some embodiments (1) stores a set of parameter values computed and sent by a first set of machines, and (2) distributes the collected parameter values to a second set of machines once it has collected the set of parameter values from all the machines in the first set. The first and second sets of machines are the same set of machines in some embodiments, while they are different sets of machines (e.g., one set has at least one machine that is not in the other set) in other embodiments. In some embodiments, the parameter collecting circuit performs computations on the parameter values that it collects and distributes the result of the computations once it has processed all the parameter values distributed by the first set of machines. The computations are aggregating operations (e.g., adding, averaging, etc.) that combine corresponding subset of parameter values distributed by the first set of machines. ## Claims ### Claim 1 A data plane circuit for a forwarding element that forwards data messages in a network connecting a plurality of machines that perform distributed computing operations, the data plane comprising: a plurality of programmable message-processing stages to perform data message forwarding operations on data tuples associated with data messages received by the data-plane circuit in order to forward the data messages within the network, wherein the plurality of programmable message-processing stages comprise a first and second set of programmable message-processing stages; a parameter extracting circuit to extract, from a subset of the data messages, parameter values computed and distributed by a first set of machines; a computation circuit to perform computations on the extracted parameter values; and a parameter forwarding circuit to send results of the computations to a second set of machines, wherein the second set of programmable message-processing stages is to implement the parameter extracting circuit, the computation circuit and the parameter forwarding circuit. ### Claim 2 (depends on 1) The data plane circuit of claim 1 further comprising a set of one or more storages in which the computation circuit is to store the results of the computations. ### Claim 3 (depends on 2) The data plane circuit of claim 2, wherein the computation circuit is to aggregate computations to combine the parameter values distributed by the first set of machines, and wherein when the parameter extracting circuit extracts a set of parameter values from a particular machine of the first set of machines, the computation circuit is to retrieve, from the storage set a result of a prior aggregation computation and perform an aggregation computation based on the retrieved result and the extracted parameter value set from the particular machine of the first set of machines. ### Claim 4 (depends on 3) The data plane circuit of claim 3, wherein the aggregation computation comprises an addition. ### Claim 5 (depends on 1) The data plane circuit of claim 1, wherein the extracted parameter values are in a floating-point format, the data-plane circuit further comprising: a floating-point to fixed-point converter to convert extracted floating-point parameter values to fixed-point parameter values on which the computation circuit is to perform its computations; and a fixed-point to floating-point converter to convert results of the computations from a fixed-point format to a floating-point format before the results are embedded in data messages to be sent to the second set of machines. ### Claim 6 (depends on 1) The data plane circuit of claim 1, wherein the parameter forwarding circuit is to retrieve the results after the data plane circuit has received parameter values from each of the first set of machines, and the computation circuit has performed computations based on the parameter values from all of the first set of machines. ### Claim 7 (depends on 6) The data plane circuit of claim 6, wherein the parameter forwarding circuit comprises a data message generator to generate a plurality of data messages to store the retrieved results to send to different machines in the second set of machines. ### Claim 8 (depends on 7) The data plane circuit of claim 7, wherein the data message generator is to generate data messages by replication of a last data message that provides a last set of one or more parameter values to complete a group of parameter values collected from the first set of machines. ### Claim 9 (depends on 8) The data plane circuit of claim 8 further comprising: a plurality of ingress message-processing pipelines, at least one comprising a plurality of programmable message-processing stages; a plurality of egress message-processing pipelines, at least one comprising a plurality of programmable message-processing stages; a traffic manager to forward data messages between ingress pipelines and egress pipelines, the traffic manager to implement the data message generator and replication of the last data message. ### Claim 10 (depends on 2) The data plane circuit of claim 2, wherein a subset of the programmable message-processing stages is programmed to implement the parameter extracting and parameter forwarding circuits and the set of one or more storages. ### Claim 11 A non-transitory machine readable medium storing a program for configuring a data-plane circuit for a forwarding element that forwards data messages in a network connecting a plurality of machines that perform distributed computing operations, the program for execution by at least one processing unit, the program comprising sets of instructions for: configuring a plurality of programmable message-processing stages to perform data message forwarding operations on data tuples associated with data messages received by the data-plane circuit in order to forward the data messages within the network; configuring a parameter extracting circuit to extract, from a subset of the data messages, parameter values computed and distributed by a first set of machines; configuring a computation circuit to perform computations on the extracted parameter values; and configuring parameter forwarding circuit to send results of the computations to a second set of machines, wherein the plurality of programmable message-processing stages comprises a first set of programmable message-processing stages and a second set of programmable message-processing stages and configuring the parameter extracting circuit, the computation circuit and the parameter forwarding circuit comprises configuring the second set of programmable message-processing stages to implement the parameter extracting circuit, the computation circuit and the parameter forwarding circuit. ### Claim 12 (depends on 11) The non-transitory machine readable medium of claim 11, wherein the program further comprises a set of instructions for identifying a set of one or more storages in which the computation circuit stores the results of the computation. ### Claim 13 (depends on 12) The non-transitory machine readable medium of claim 12, wherein the computation circuit performs aggregating computations to combine the parameter values distributed by the first set of machines, wherein when the parameter extracting circuit extracts a set of parameter values from a particular first set machine, the computation circuit retrieves from the storage set a result of a prior aggregating computation and performs an aggregating computation based on the retrieved result and the extracted parameter value set from the particular first set of machines. ### Claim 14 (depends on 11) The non-transitory machine readable medium of claim 11, wherein the message-processing stages of the first and second sets of programmable message-processing stages comprise match-action circuits that attempt to match attribute sets associated with the data messages to records in match tables and perform actions associated with match table records that match the attribute sets of the data messages. ### Claim 15 (depends on 11) The non-transitory machine readable medium of claim 11, wherein at least a set of programmable message-processing stages comprises stateful processing units, and the program comprises a set of instructions for configuring the computation circuit to store computation results in a set of the stateful processing units. ### Claim 16 (depends on 11) The non-transitory machine readable medium of claim 11, wherein the extracted parameter values are in a floating-point format, wherein the program further comprises sets of instructions for: configuring a floating-point to fixed-point converter to convert extracted floating-point parameter values to fixed-point parameter values on which the computation circuit performs its computations; and configuring a fixed-point to floating-point converter to convert the results to fixed-point format before the results are sent to the second set of machines. ### Claim 17 (depends on 11) The non-transitory machine readable medium of claim 11, wherein the parameter forwarding circuit retrieves the results after the data plane circuit has received parameter values from each of the first set of machines, and the computation circuit has performed computations based on the parameter values from all of the first set of machines. ### Claim 18 (depends on 17) The non-transitory machine readable medium of claim 17, wherein the set of instructions for configuring a parameter forwarding circuit comprises a set of instructions for configuring a data message generator to generate a plurality of data messages to store the retrieved results to send to different machines in the second set of machines. ### Claim 19 (depends on 18) The non-transitory machine readable medium of claim 18, wherein the set of instructions for configuring the data message generator comprises a set of instructions for replicating a last data message that provides a last set of one or more parameter values to complete a group of parameter values collected from the first set of machines. ### Claim 20 (depends on 11) The non-transitory machine readable medium of claim 11, wherein the first and second set of machines are the same set of machines. ### Claim 21 (depends on 11) The non-transitory machine readable medium of claim 11, wherein one set of machines has at least one machine that is not in the other set of machines.