System and method of executing neural networks

A method of training a NN on one or more target computing devices, the NN comprising a plurality of layers and at least one layer comprising one or more kernels, the method comprising:

• receiving a data structure representing the NN;
• receiving cache parameter values pertaining to cache memory of the one or more target computing devices;
• analyzing the data structure to produce one or more tasks, each task comprising at least one computation that pertains to a kernel of the NN;
• selecting a sparse version of one or more kernels of the one or more tasks and replacing the at least one kernel with the sparse version;
• compiling the one or more tasks to produce one or more respective tensor columns comprising task instruction code representing the at least one computation,
• wherein at least one tensor column comprises task instruction code that represents at least one computation spanning a plurality of layers of the NN;
• storing the one or more tensor columns in respective cache memories of the one or more target computing devices according to the received cache parameter values;
• and
• training the NN on incoming training data by executing the task instruction code on the incoming data, within the cache memory space.

A system for inferring a NN on one or more target computing devices, the system comprising:

• a non-transitory memory device, wherein modules of instruction code are stored, and a processor associated with the memory device, and configured to execute the modules of instruction code, whereupon execution of said modules of instruction code, the processor is configured to:
• receive a data structure representing the NN;
• receive cache parameter values pertaining to cache memory of the one or more target computing devices;
• analyze the data structure to produce one or more tasks, each task comprising at least one computation that pertains to a kernel of the NN;
• select a sparse version of at least one kernel of the one or more tasks and replacing the at least one kernel with the sparse version;
• and
• compile the one or more tasks to produce one or more respective tensor columns comprising task instruction code representing the at least one computation,
• wherein the one or more tensor columns are adapted to fit in respective one or more cache memories of the one or more target computing devices, according to the received cache parameter values, wherein at least one tensor column comprises task instruction code that represents at least one computation spanning a plurality of layers of the NN.