Systems and methods for improved neural network execution
AssigneeNeuralmagic Inc.
InventorsAlexander Matveev, Nir Shavit
Life of this patent
Estimating the term…
Abstract
A method and system for computing one or more outputs of a neural network having a plurality of layers is provided. The method and system can include determining a plurality of sub-computations from total computations of the neural network to execute in parallel wherein the computations to execute in parallel involve computations from multiple layers. The method and system also can also include avoiding repeating overlapped computations and/or multiple memory reads and writes during execution.
Claims20
A method for computing one or more outputs of a neural network having a plurality of layers, the method comprising:
- analyzing, by a computing device, the neural network to determine a plurality of sub-computations from total computations of the neural network, wherein determining each sub-computation comprises:
- i) determining a group of outputs for each sub-computation;
- ii) for each group of outputs, determining a largest number of inputs that are necessary to compute the group of outputs and results in a memory requirement less than a threshold;
- and
- computing the outputs of the neural network by executing each sub-computation.
A system comprising an improved convolutional neural network (CNN), the CNN comprising:
- a memory;
- and
- a processor;
- the processor configured to:
- analyzing, by a computing device, the neural network to determine a plurality of sub-computations from total computations of the neural network, wherein determining each sub-computation comprises:
- iii) determining a group of outputs for each sub-computation;
- iv) for each group of outputs, determining a largest number of inputs that are necessary to compute the group of outputs and results in a memory requirement less than a threshold;
- and
- computing the outputs of the neural network by executing each sub-computation.
A non-transitory computer readable storage medium storing a set of instructions for causing a computer to generate a performance rating for a developer, by performing the operations of:
- analyzing a neural network to determine a plurality of sub-computations from total computations of the neural network, wherein determining each sub-computation comprises:
- v) determining a group of outputs for each sub-computation;
- vi) for each group of outputs, determining a largest number of inputs that are necessary to compute the group of outputs and results in a memory requirement less than a threshold;
- and
- computing the outputs of the neural network by executing each sub-computation.
Dependencies are inferred from the claim text.