Accelerating distributed reinforcement learning with in-switch computing

An accelerator comprising:

• a decoding subsystem to parse a packet header and a payload from an incoming packet, wherein the incoming packet is part of gradient vectors received from worker computing devices, and the payload comprises a gradient segment that is assigned a segment number by the decoding subsystem;
• an aggregation subsystem coupled to the decoding subsystem, the aggregation subsystem to aggregate gradient data and buffer the aggregated gradient data of the gradient segment as incoming packets of the gradient vectors are received;
• a segment counter of the aggregation subsystem to track progress of aggregation of the aggregated gradient data according to the segment number;
• and
• an output subsystem coupled to the aggregation subsystem and the segment counter, the output subsystem to concatenate and output an aggregated data packet of the aggregated gradient data for the gradient segment in response to the segment counter indicating the gradient segment has been completely aggregated.

A method comprising:

• analyzing, by an input arbiter, packet headers of incoming packets of a programmable switch to determine which of the incoming packets are part of gradient vectors received from worker computing devices that are performing reinforcement learning as a group;
• transferring, by the input arbiter, the incoming packets to an accelerator coupled to the input arbiter;
• assigning, by the accelerator, a segment number to a gradient segment of the gradient vectors, wherein the gradient segment is a corresponding sub-portion of each respective gradient vector of the gradient vectors;
• aggregating and buffering, by the accelerator, gradient data of the incoming packets that is associated with the segment number;
• tracking buffering of the aggregated gradient data according to the segment number by incrementing a segment counter upon buffering gradient data from each incoming packet associated with a corresponding worker computing device of the worker computing devices;
• responsive to the segment counter reaching a threshold number, generating an aggregated data packet containing the aggregated gradient data associated with the gradient segment of the gradient vectors;
• and
• transferring, by the accelerator, the aggregated data packet to the input arbiter to be forwarded back to the worker computing devices.