Systems and methods for vectorized fft for multi-dimensional convolution operations

A hardware-based programmable deep learning processor (DLP), comprising:

• an on-system memory (OSM) and one or more controllers configured to access a plurality of external memory resources via direct memory access (DMA);
• a plurality of programmable tensor engines configured to perform a plurality of convolution operations by applying one or more kernels on multi-dimensional input data to generate deep learning processing results for pattern recognition and classification based on a neural network, wherein each of the plurality of tensor engines further comprises:
• one or more vector processing engines each configured to:
• vectorize the multi-dimensional input data at each layer of the neural network to generate a plurality of vectors;
• perform multi-dimensional fast Fourier transform (FFT) on the generated vectors and/or the kernels to create output for the convolution operations;
• wherein one or more of the plurality of convolution operations are each partitioned among the plurality of vector processing engines, wherein each of the plurality of vector processing engines is configured to perform a sub-task of each of the one or more of the plurality of convolution operations in parallel.

A hardware-based programmable deep learning processor (DLP), comprising:

• an on-system memory (OSM) and one or more controllers configured to access a plurality of external memory resources via direct memory access (DMA);
• a plurality of programmable tensor engines configured to perform a plurality of convolution operations by applying one or more kernels on multi-dimensional input data to generate deep learning processing results for pattern recognition and classification based on a neural network, wherein each of the plurality of tensor engines further comprises:
• one or more vector processing engines each configured to:
• vectorize the multi-dimensional input data at each layer of the neural network to generate a plurality of vectors;
• perform multi-dimensional fast Fourier transform (FFT) on the generated vectors and/or the kernels to create output for the convolution operations;
• a programmable CPU having its own instruction cache and data cache configured to store a plurality of instructions from a host and the retrieved data from the OSM and/or the external memory resources, respectively.

A method to support hardware-based programmable vectorized fast Fourier transform (FFT) for multidimensional convolution, comprising:

• prefetching multi-dimensional input data and/or one or more kernels from an on system memory (OSM) and/or a plurality of external memory resources via direct memory access (DMA);
• storing the prefetched input data from the OSM and/or the external memory resources in an online memory (OLM);
• vectorizing the multi-dimensional input data to generate a plurality of vectors at each layer of a neural network used for a plurality of convolution operations;
• performing multi-dimensional FFT on the generated vectors and/or the kernels to create output for the convolution operations;
• outputting deep learning processing results for pattern recognition and classification to a host based on the output for the convolution operations.

A method to support hardware-based programmable vectorized fast Fourier transform (FFT) for multidimensional convolution, comprising:

• partitioning each of one or more of a plurality of convolution operations among a plurality of vector processing engines, wherein each of the plurality of vector processing engines is configured to perform a sub-task of each of the one or more of the plurality of convolution operations in parallel;
• vectorizing multi-dimensional input data to generate a plurality of vectors at each layer of a neural network used for the plurality of convolution operations;
• performing multi-dimensional FFT on the generated vectors and/or the kernels to create output for the convolution operations;
• outputting deep learning processing results for pattern recognition and classification to a host based on the output for the convolution operations.

A method to support hardware-based programmable vectorized fast Fourier transform (FFT) for multidimensional convolution, comprising:

• accepting a plurality of instructions from a host and submitting the instructions to program a plurality of vector processing engines for the vectorized FFT for multidimensional convolution;
• vectorizing multi-dimensional input data to generate a plurality of vectors at each layer of a neural network used for a plurality of convolution operations;
• performing multi-dimensional FFT on the generated vectors and/or one or more kernels to create output for the convolution operations;
• outputting deep learning processing results for pattern recognition and classification to a host based on the output for the convolution operations.