Method and apparatus with neural network convolution operation

A processor-implemented neural network method comprising:

• using one or more processors, generating at least one result of a neural network, which comprises a plurality of layers, based on an input to the neural network and a propagating of the input through one or more of the plurality of layers based on respective performances of neural network operations between the propagated input and respective parameters of the one or more of the plurality of layers, wherein the respective performances of the neural network operations include:
• generating a first output line of an output feature map of a layer, of the plurality of layers, by performing a convolution operation of the layer between a first input line group, comprising a plurality of first input lines that each include first vectors, of an input feature map and weight element of different weight kernels;
• and
• generating a first output of an operation block, which includes at least the convolution operation of the layer, based on the first output line;
• storing the first output in a memory in which the input feature map is stored by performing a first overwriting of the first output to a memory space, of the memory, of at least one input line of the input feature map;
• generating a second output line of the output feature map of the layer by performing another convolution operation of the layer based on a second input line group, comprising a plurality of second input lines that each include second vectors, of the input feature map and the different weight kernels;
• generating a second output of the operation block based on the second output line;
• and
• storing the second output in the memory in which the input feature map is stored by performing a second overwriting of the second output to another memory space, of the memory, of at least one other input line of the input feature map,
• wherein, the input feature map comprising the first input line group and the second input line group and the output feature map comprising the first output line and the second output line is stored in the memory in a line data structure,
• wherein the convolution operation is repeatedly performed until an index of a current input line is equal to the total number of the plurality of first input lines,
• wherein the storing of the second output is performed subsequent to the first overwriting,
• wherein each of the first input line group and the second input line group comprises a number of input lines corresponding to a height of the weight kernels, and the height of the weight kernels is less than a height of the input feature map,
• wherein the line data structure is configured to enable the first and second overwriting by organizing the first and second output lines in a form of a 3D block with the data ordering applied in an order of the width direction, the channel direction, and the height direction, such that each overwriting operation aligns overwritten memory spaces with spanning of each of the first and second output line vectors in a width direction of the 3D block, and
• wherein the line data structure is further configured to enable simultaneous single-instruction multiple-data (SIMD) processing of multiple channels by overwriting only memory locations containing input lines unused in subsequent convolution operations.

A neural network apparatus comprising:

• one or more processors configured to generate at least one result of a neural network, which comprises a plurality of layers, based on an input to the neural network and a propagating of the input through one or more of the plurality of layers based on respective performances of neural network operations between the propagated input and respective parameters of the one or more of the plurality of layers, wherein the respective performances of the neural network operations include:
• a generation of a first output line of an output feature map of a layer, of the plurality of layers, by performing a convolution operation of the layer between a first input line group of an input feature map and weight elements of different weight kernels, with the first output line including a plurality of first output line vectors that are respectively dependent on the different weight kernels, to respectively correspond to different output channels of the output feature map;
• a generation of a first output of an operation block, which includes at least the convolution operation of the layer, based on the first output line vectors of the first output line, and
• a storage of the first output in a memory in which the input feature map is stored by performing a first overwriting of the first output to a memory space, of the memory, of at least one input line of the input feature map;
• a generation of a second output line of the output feature map of the layer by performing another convolution operation of the layer based on a second input line group of the input feature map and the different weight kernels, with the second output line including a plurality of second output line vectors that are respectively dependent on the different weight kernels, to respectively correspond to the different output channels of the output feature map;
• a generation of a second output of the operation block based on the second output line vectors of the second output line;
• and
• a storage of the second output in the memory in which the input feature map is stored by performing a second overwriting of the second output to another memory space, of the memory, of at least one other input line of the input feature map,
• wherein the convolution operation is repeatedly performed until an index of a current input line is equal to the total number of input lines of the first input line group,
• wherein the storing of the second output is performed subsequent to the first overwriting,
• wherein each of the first input line group and the second input line group comprises a number of input lines corresponding to a height of the weight kernels, and the height of the weight kernels is less than a height of the input feature map,
• wherein, the input feature map comprising the first input line group and the second input line group and the output feature map comprising the first output line and the second output line is stored in the memory in a line data structure,
• wherein the line data structure is configured to enable the first and second overwriting by organizing the first and second output lines in a form of a 3D block with the data ordering applied in an order of the width direction, the channel direction, and the height direction, such that each overwriting operation aligns overwritten memory spaces with spanning of each of the first and second output line vectors in a width direction of the 3D block, and
• wherein the line data structure is further configured to enable simultaneous single-instruction multiple-data (SIMD) processing of multiple channels by overwriting only memory locations containing input lines unused in subsequent convolution operations.