System and method of determining and executing deep tensor columns in neural networks
AssigneeNeuralmagic Inc.
InventorsAlexander Matveev, Nir Shavit, Govind Ramnarayan, Tyler Michael Smith, Sage Moore
Life of this patent
Estimating the term…
Abstract
A system and method may partition and/or execute a NN, by, for a graph including nodes and hyper edges, each node representing a data item in the NN and each hyper edge representing an operation in the NN, identifying a deep tensor column comprising a subset of the nodes and a subset of the hyper edges, such that the operations in the deep tensor column, when executed, use only data which fits within a preselected cache.
Claims20
A method of partitioning a neural network (NN), the method comprising:
- receiving a representation of a NN comprising nodes and edges;
- and
- identifying a deep tensor column comprising a subset of the nodes and a subset of the edges, such that the operations in the deep tensor column, when executed, use only data which fits within a preselected cache and wherein identifying a deep tensor column comprises using a cost function measured by the compute to memory ratio of the deep tensor column, the compute to memory ratio determining a measure of the ratio of compute to memory accesses.
A system for partitioning a neural network (NN), the system comprising:
- a memory;
- and
- a processor to:
- receive a NN comprising nodes and edges;
- and
- identify a deep tensor column comprising a subset of the nodes and a subset of the edges, such that the operations in the deep tensor column, when executed, use only data which fits within a preselected cache;
- wherein identifying a deep tensor column comprises using a cost function measured by the compute to memory ratio of the deep tensor column, the compute to memory ratio determining a measure of the ratio of compute to memory accesses.
A method of executing a neural network (NN), the method comprising:
- receiving a set NN partitions, each NN partition comprising nodes and edges, such that the operations in each partition, when executed, use only data which fits within a preselected cache, wherein the NN partitions are identified using a cost function measured by the compute to memory ratio of a partition, the compute to memory ratio determining a measure of the ratio of compute to memory accesses;
- and
- executing the plurality of NN partitions in order to execute the NN.
Dependencies are inferred from the claim text.