Acceleration of neural networks with stacks of convolutional layers

A computer-implemented method comprising:

• receiving a neural network model for a neural network, the neural network being configured to generate an output dataset from an input dataset using convolution operations performed by at least a first computing engine (CE) and a second CE, wherein the first CE and the second CE are part of a neural network processor, and wherein the input dataset is to be transferred to the neural network processor from an external memory and comprises a first chunk and a second chunk;
• and
• generating, based on the neural network model, instructions for executing by the first CE and the second CE in parallel to:
• process a last portion of a respective chunk to compute respective shared states;
• process a middle portion of the respective chunk to generate a first sub-portion of a respective portion of the output dataset;
• receive, while processing the middle portion of the respective chunk, additional shared states for processing, wherein:
• the first chunk is the respective chunk for the first CE,
• the second chunk is the respective chunk for the second CE,
• the last portion of the first chunk overlaps with a first portion of the second chunk,
• the additional shared states received by the second CE are the respective shared states computed by the first CE and are obtained from a local memory of the first CE,
• the second CE receives the respective shared states computed by the first CE as a substitute for intermediate states that would otherwise have been computed by the second CE using the first portion of the second chunk, and
• the first CE receives pre-computed states as the additional shared states;
• and
• generate a second sub-portion of the respective portion of the output dataset using the received additional shared states.

A non-transitory computer readable medium having stored therein instructions that, when executed by one or more processors cause the one or more processors to execute a compiler, the compiler performing operations including:

• receiving a neural network model for a neural network, the neural network being configured to generate an output dataset from an input dataset using convolution operations performed by at least a first computing engine (CE) and a second CE, wherein the first CE and the second CE are part of a neural network processor, and wherein the input dataset is to be transferred to the neural network processor from an external memory and comprises a first chunk and a second chunk;
• and
• generating, based on the neural network model, instructions for executing by the first CE and the second CE in parallel to:
• process a last portion of a respective chunk to compute respective shared states;
• process a middle portion of the respective chunk to generate a first sub-portion of a respective portion of the output dataset;
• receive, while processing the middle portion of the respective chunk, additional shared states for processing, wherein:
• the first chunk is the respective chunk for the first CE,
• the second chunk is the respective chunk for the second CE,
• the last portion of the first chunk overlaps with a first portion of the second chunk,
• the additional shared states received by the second CE are the respective shared states computed by the first CE and are obtained from a local memory of the first CE,
• the second CE receives the respective shared states computed by the first CE as a substitute for intermediate states that would otherwise have been computed by the second CE using the first portion of the second chunk, and
• the first CE receives pre-computed states as the additional shared states;
• and
• generate a second sub-portion of the respective portion of the output dataset using the received additional shared states.

A method comprising:

• receiving, by a first computing engine (CE) and a second CE in a neural network processor, an input dataset divided into chunks comprising a first chunk and a second chunk, each chunk to be processed by a respective CE to generate a respective portion of an output dataset using convolution operations, wherein:
• the input dataset is transferred to the neural network processor from an external memory,
• the first chunk and the second chunk each include a first portion, a middle portion, and a last portion,
• the first chunk is the respective chunk for the first CE,
• the second chunk is the respective chunk for the second CE, and
• the last portion of the first chunk overlaps with the first portion of the second chunk;
• processing, by each of the first CE and the second CE, the last portion of the respective chunk to compute respective shared states;
• processing, by each of the first CE and the second CE, the middle portion of the respective chunk to generate a first sub-portion of the respective portion of the output dataset;
• receiving, by each of the first CE and the second CE and while processing the middle portion of the respective chunk, additional shared states for processing, wherein:
• the additional shared states received by the second CE are the respective shared states computed by the first CE and are obtained from a local memory of the first CE,
• the second CE receives the respective shared states computed by the first CE as a substitute for intermediate states that would otherwise have been computed by the second CE using the first portion of the second chunk, and
• the additional shared states received by the first CE are pre-computed states;
• and
• generating, by each of the first CE and the second CE, a second sub-portion of the respective portion of the output dataset using the received additional shared states.