Efficient scaling of partitioned neural network inference
AssigneeCisco Technology, Inc.
InventorsMyungjin Lee, Jayanth SRINIVASA, Ali PAYANI, Ramana Rao V.R. KOMPELLA
Life of this patent
Estimating the term…
Abstract
In one implementation, a controller determines performance of a partitioned neural network. The controller identifies, based on the performance, a particular partition of the partitioned neural network as a bottleneck. The controller configures a first device to execute a replica of the particular partition. The controller configures a multiplexer that provides an output of the particular partition or the replica of the particular partition as input to a downstream partition of the partitioned neural network.
Claims20
A method comprising:
- determining, by a controller, performance of a partitioned neural network;
- identifying, by the controller and based on the performance, a particular partition of the partitioned neural network as a bottleneck;
- configuring, by the controller, a first device to execute a replica of the particular partition;
- and
- configuring, by the controller, a multiplexer that provides an output of the particular partition or the replica of the particular partition as input to a downstream partition of the partitioned neural network.
An apparatus, comprising:
- a network interface to communicate with a computer network;
- a processor coupled to the network interface and configured to execute one or more processes;
- and
- a memory configured to store a process that is executed by the processor, the process when executed configured to:
- determine performance of a partitioned neural network;
- identify, based on the performance, a particular partition of the partitioned neural network as a bottleneck;
- configure a first device to execute a replica of the particular partition;
- and
- configure a multiplexer that provides an output of the particular partition or the replica of the particular partition as input to a downstream partition of the partitioned neural network.
A tangible, non-transitory, computer-readable medium storing program instructions that cause a controller to execute a process comprising:
- determining, by a controller, performance of a partitioned neural network;
- identifying, by the controller and based on the performance, a particular partition of the partitioned neural network as a bottleneck;
- configuring, by the controller, a first device to execute a replica of the particular partition;
- and
- configuring, by the controller, a multiplexer that provides an output of the particular partition or the replica of the particular partition as input to a downstream partition of the partitioned neural network.
Dependencies are inferred from the claim text.