Methods and Apparatuses for Bottleneck Stages in Neural-Network Processing

A hardware accelerator circuit configured as a bottleneck stage for neural-network processing and comprising:

• convolution circuitry comprising a data path that includes a series of first, second, and third convolution stages;
• memory circuitry comprising first, second, and third stage-input buffers for the first, second, and third convolution stages, along with an output data buffer and a weight buffer;
• and
• control circuitry and associated bus interface circuitry, the control circuitry configured to read input data vectors in from an external memory into the first stage-input buffer and to write corresponding output data vectors from the output data buffer back to the external memory;
• wherein the first convolution stage comprises a 1×1×M convolution circuit that is configured to use respective weights from the weight buffer to produce first convolution results for respective input data vectors held in the first stage-input buffer, each input data vector being one among a set of input data vectors to be processed by the hardware accelerator circuit and comprising a stack of N channel values corresponding to a respective grid position of an input data grid associated with the set of input data vectors, the first convolution results for each input vector being an expanded vector of M channel values, where M>N;
• wherein the second convolution stage comprises a depth-wise R×R×1 convolution circuit that is configured to use respective weights from the weight buffer to produce second convolution results for respective R×R subsets of the expanded vectors, and wherein the second stage-input buffer holds, at least incrementally, the first convolution results produced by the first convolution stage;
• wherein the third convolution stage comprises a further 1×1×S convolution circuit that is configured to use respective weights from the weight buffer to produce third convolution results from the second convolution results, wherein the third stage-input buffer holds, at least incrementally, the second convolution results produced by the second convolution stage, where third convolution results comprise compressed vectors corresponding to each input data vector being processed in a current processing cycle of the hardware accelerator circuit, each compressed vector having a length S, where S<M;
• and
• wherein the control circuitry is configured to write respective output data vectors held in the output data buffer back to the external memory, each output data vector corresponding to a respective one of the input data vectors in the set of input data vectors and comprising the corresponding compressed vector produced by the third convolution stage, or a combination of the corresponding compressed vector produced by the third convolution stage and the respective input data vector.

A method of neural-network processing, the method comprising:

• arranging channel data as input data vectors and storing the input data vectors in a memory circuit on a memory-word basis, the channel data resulting from processing original input data through one or more layers of a neural network and each input data vector corresponding to a respective grid position in a data grid associated with the original input data and comprising a stack of N channel values corresponding to N channels of data resulting from the processing;
• and
• processing the input data vectors in a hardware accelerator circuit that implements a bottleneck stage of the neural-network processing and reads in respective ones of the input data vectors from the memory circuit on the memory-word basis;
• wherein processing the input data vectors in the hardware accelerator circuit comprises:
• producing first convolution results via a first convolution stage that performs 1×1×M convolutions on the input data vectors read into the hardware accelerator circuit, where M>N;
• producing second convolution results via a second convolution stage that performs R×R×1 depth-wise convolutions of the first convolution results produced for respective subsets of the input data vectors read into the hardware accelerator circuit;
• producing third convolution results via a third convolution stage that performs 1×1×S convolutions on the second convolution results, the third convolution results being respective compressed data vectors for each subset of input data vectors read into the hardware accelerator circuit, where S<M;
• and
• writing output data vectors back to the memory circuit, or another memory circuit, wherein each output data vector corresponds to a respective one of the input data vectors in the set of input data vectors and comprises the corresponding compressed vector produced by the third convolution stage, or a combination of the corresponding compressed vector produced by the third convolution stage and the respective input data vector.