Communication-aware inference serving for partitioned neural networks
AssigneeCisco Technology, Inc.
InventorsMyungjin Lee, Gustav Adrian Baumgart, Jaemin Shin, Ramana Rao V.R. Kompella
Life of this patent
Estimating the term…
Abstract
In one implementation, a device generates outputs of nodes in a upstream layer of a partitioned neural network. The device assigns priorities to each of the outputs of the nodes. The device selects, based on the priorities, a subset of the outputs to send to a remote device. The device sends, via a computer network, the subset of the outputs to the remote device for input to a downstream layer of the partitioned neural network.
Claims20
A method comprising:
- generating, by a device, outputs of nodes in an upstream layer of a partitioned neural network;
- assigning, by device, priorities to each of the outputs of the nodes;
- selecting, by the device and based on the priorities, a subset of the outputs to send to a remote device;
- and
- sending, by the device and via a computer network, the subset of the outputs to the remote device for input to a downstream layer of the partitioned neural network.
An apparatus, comprising:
- a network interface to communicate with a computer network;
- a processor coupled to the network interface and configured to execute one or more processes;
- and
- a memory configured to store a process that is executed by the processor, the process when executed configured to:
- generate outputs of nodes in an upstream layer of a partitioned neural network;
- assign priorities to each of the outputs of the nodes;
- select, based on the priorities, a subset of the outputs to send to a remote device;
- and
- send, via a computer network, the subset of the outputs to the remote device for input to a downstream layer of the partitioned neural network.
A tangible, non-transitory, computer-readable medium storing program instructions that cause a device in a computer network to execute a process comprising:
- generating, by the device, outputs of nodes in an upstream layer of a partitioned neural network;
- assigning, by device, priorities to each of the outputs of the nodes;
- selecting, by the device and based on the priorities, a subset of the outputs to send to a remote device;
- and
- sending, by the device and via the computer network, the subset of the outputs to the remote device for input to a downstream layer of the partitioned neural network.
Dependencies are inferred from the claim text.